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Over the past decade, there has been a dramatic growth in the creation of community collaborations to design 
and implement new service approaches for children, youth, and families to better address individual, 
neighborhood, and community needs. Many of these community collaborations have become very 
sophisticated in their work, learning and expanding their vision as they go forward. 

Many began primarily as collaborations of service providers — involving organizations and agencies providing 
health, education, and human services. Increasingly, however, such collaborations have moved beyond a 
strictly service approach, recognizing that they must involve the entire community to succeed and that they 
must address economic and social as well as human capital development needs. 

As collaborations seek to improve child, youth, and family outcomes on a community level, they inevitably 
recognize the complexity, as well as the importance, of their task. 

In 1994, three organizations -- the Center for the Study of Social Policy, the Child and Family Policy Center, 
and the Family Resource Coalition of America — came together with the hope of combining our experiences 
in the field into something that would be useful to community collaborations seeking comprehensive change. 
The Center for the Study of Social Policy initiated this work a part of its Improved Outcomes for Children 
project. The Academy for Educational Development/Center for Youth Development and Policy Research 
provided guidance and support to us in recognizing the importance of youth development and involvement. 

Organizing our experiences and the knowledge we have gained has proved to be a challenge. We started from 
the premise that society can improve the well-being of its children, but to do so will require fundamental changes 
in the way communities (both their public and private systems) support and serve children, youth, and families. 
The work over the last decade represents a base upon which to build. 

If we are to impact the current sobering outcomes and trends threatening our children and our society, however, 
the next decade will require both a broadening and a deepening of these reform efforts. We will have to break 
new ground to succeed. 
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Our experience tells us that the changes needed to improve the well-being of children, youth, and families are 
broad-reaching and involve three complementary and interrelated elements: 

1. Effective services and supports that reach out to and connect with children, youth, and families; 

2. Community and economic development that provides opportunities for sustaining and supporting 
families through employment within all neighborhoods in the community; and 

3. Neighborhood and consumer participation and involvement in constructing those services and 
supports and other conditions required to ensure sound futures for children and communities. 

This requires changes on many levels. In the end, we sought to arrange our knowledge base about successful 
reform efforts into different subject areas. While the goal is to produce a comprehensive guide, the different 
subject areas also lend themselves to separate publication, represented by this series of Guidebooks. 

These Guidebooks share a common format. Each Guidebook begins with a brief rationale for the relevance of 
the subject area and how it fits as a vital part of the work of a community collaboration seeking comprehensive 
reform. Next is a discussion of the key ideas embodied in the subject, followed by a discussion of the steps 
involved in putting the ideas into practice. Finally, the appendices contain additional related materials for further 
discussion of the subject. 

These subject areas, of course, are not discrete. They overlap and interconnect. The back cover of this 
Guidebook provides a brief description of all the proposed chapters for the comprehensive guide and identifies 
which ones currently are available in Guidebook form. 

The National Center for Service Integration Clearinghouse is responsible for editing and publishing these 
Guidebooks. The preparation of the Guidebooks has been supported by the Annie E. Casey Foundation. 

The three sponsoring organizations are excited about the work of community collaborations across the country. 
We believe that this work holds the best promise for truly addressing the needs of our children, youth, and 
families. As the adjoining page suggests,, we believe it is a matter of commitment and now is the time for action. 



Center for the Study of Social Policy 
Child and Family Policy Center 
Family Resource Coalition of America 
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x I Matter of Commitment: 
Making the Case for Reform 

1. Things are getting serious. Current systems fail too many children, youth, and families, and trends in 
child well-being are deteriorating. This places increasing numbers of children, youth, and families at risk and 
weakens society as a whole. 

2. We know what works (but we are doing it only at the margins). The country abounds with 
promising programs, services, and strategies that are helping children, youth, and families succeed. They embrace 
new principles of effective practice and emphasize neighborhood-based approaches that build local systems of 
support, but have had little effect on a community level. In part, this is because these efforts are being implemented 
only on a small scale and at the margin, without transforming larger public system responses. 

3. Implementing what works at the community level requires changes in all systems. Isolated 
programs, services, and supports fight against the odds to help children, youth, and families. All systems must 
change and respond according to new principles of effective practice, including such mainstream systems as 
education, health services, and the array of human services and incorporating housing, job training, and economic 
development activities. 

4. Everyone has a stake and a responsibility. Achieving success at a community level requires new 
partnerships and collaborations - within and across public systems, at all levels of government, in publicly- 
financed services and in voluntary community organizations. Most importantly, it requires involvement of the 
youth and families whose futures are most at stake. The diversity of perspectives within the community need to 
be represented in the decision-making process. The business community and the faith community, as well as many 
other interests, need to support and help guide the work. Every part of the community has a stake and a role to play. 

5. We can succeed; it’s a matter of commitment. Although the path to success is still being cleared and 
constructed, the journey is not hopeless. It is simply a matter of sustained commitment to achieve that success. 
Moreover, there is no other way to get to where we need to go. A small but representative group of truly committed 
people can build the commitment needed among others for the journey. The time to start is now. 
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The vision for comprehensive community reform starts with the goal of improving the 
well-being of children and youth, their families, and their neighborhoods and communities. 
All aspects of this reform process lead back to the goal of improving outcomes. 

The establishment of community outcomes can serve as a driving force for reform by 
making the community aware of the need for change. This, however, is only one of 
three reasons for establishing community outcomes. Used well, community outcomes 
also can help guide the development of effective strategies for achieving them, many 
of which will cut across traditional professional service domains. Finally, community 
outcomes can form the basis for establishing new systems of accountability and assuring 
that community outcomes are achieved. 

This Guidebook discusses the importance of building consensus in establishing 
outcomes to use as a driving force for reform. Community collaboratives have the 
capacity to develop a set of measurable outcome indicators and performance measures 
that can be use to inform and guide reform efforts and hold themselves accountable to 
achieving improved outcomes. 



Guidebook 2 
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# %ey Ideas 

The Case for Using Outcomes to Guide Reforms 

There is growing recognition that children and youth, and therefore society, are at 
increasing risk. The well-being of many of America's children and youth is worsening. 
Whether implicitly or explicitly, most community collaboratives form because the current 
way systems operate is failing to achieve desired goals for children, for their families, 
or for the neighborhoods and communities in which they reside. 

Simply identifying and publicizing the current status of children, youth, and families can 
serve as an impetus for reform. It can help produce the needed sense of urgency to 
take action. Table 2-1 provides select data showing that, even during the relative 
prosperity and strong business optimism of the last decade, the well-being of children 
and families has declined on many important indicators. 

Many reports and statistical summaries have shown that children, youth, and families 
are facing more difficulties in the areas of physical health, school success, and safety 
in the home and in the community. The rates of adolescent parenting and juvenile 
delinquency have skyrocketed, although these rates have experienced small declines 
in the last couple of years. Family economic and social stability is also threatened. 
When particular groups or geographic populations are examined, there also is evidence 
of wide disparity in well-being along these dimensions. A number of national reports 
(several of which are referenced in Appendix 2-1) provide detailed information about 
these trends. 
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Table 

A Tale of Two Trends : ||||| 



Since December 3 1 , 1987 

■The Gross Domestic Product (GDP) has risen 54.6% ($4,692 billion in 1 987 to $7,254 
billion in 1 995), 17.1% when adjusted for inflation 

■The Dow Jones Industrial Average has more than tripled from 1 950. 1 0 at the end 
of 1 987 to 6448.27 at the end of I 996) 

■The mean real (inflation adjusted) income of the highest 20% of American 
households has increased by 7.9% (from $ 1 0 1 ,448 in 1 988 to $ 1 09,424 in 1 995, 
measured in 1 995 dollars 

Between 1 987 and 1 994-5 

■ Low birthweight has increased 5.8% (6.9% in 1 987 to 7.3% in 1 994) 

■ Teen birth rate has increased 1 8.8% (3.2% of 1 5-1 7 year-old females in 1 987 to 
3.8% in 1994) 

■ Juvenile violent crime arrest rate has increased 58.6% (3 1 9 arrests per 1 00,000 1 0- 
I 7 year-olds in 1 987 to 506 in 1 993) 

■ Children in poverty rate has increased 4.0% (20% in 1 987 to 20.8% in 1 995) 

■ Young children (0-6) in extreme poverty rate has increased I 1 .9% ( 1 0.0% in 1 987 
to I 1 .9% in 1994) 

■ Families headed by single parent rate has increased 16.1% (22.3% in 1987 to 
25.9% in 1995) 

■ The mean real income of the lowest 80% of American households has declined 
2.0% (from $29,41 6 in 1 988 to $28,81 6 in 1 995 dollars) 



Appendix 2- / . Ant nutated 
bibliography of key national 
resources tracking trends in child 
and family well-being 
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The emphasis upon outcomes presented in this guidebook goes much deeper than 
simply providing a rationale for reform, however. Outcomes also can be used in a 
fundamental way to guide refomi efforts and to hold individuals, organizations, and 
strategies accountable for their role in achieving results. 

In an article in the NCS1 Resource Brief Making a Difference, Lisbcth Schorr offered 
a number of reasons for moving toward ''results-bascd' 1 accountability in order to 
guide reform activities: 

1 . Rcsults-based accountability can replace, or substantially diminish the need 
for, centralized bureaucratic micro-management and rigid rules. 

2. Outcome information can assure funders and the public that investments 
are producing results. 

3. Agreement on desired results can facilitate cross-systems collaboration on 
behalf of children. 

4. Agreement on desired results helps to minimize investment in activities that 
do not contribute to improved results. 

5. Information about results enhances community and agency capacity to judge 
the effectiveness of their efforts. 

6. A focus on results clarifies whether allocated resources are adequate to 
achieve the outcomes expected by funders. 

Two of these points are worth additional discussion. The first point recognizes that 




11 



Defining the Prize : From Agreed-Upon Outcomes to Resu/ts-Based Accountability 



current accountability systems largely arc process-based, based on adherence to rules 
and regulations or the provision of specific units of services. There is increasing 
recognition, however, that a "one size fits all"’ approach fails to effectively serve many 
families. As refonns stress more discretion and individual contouring of services and 
supports, the existing accountability system based upon adherence to rules is no longer 
possible. A more rcsults-bascd accountability system is needed to ensure wise use of 
funds and resources. 

The third point emphasizes that some outcomes being sought require actions that 
extend beyond "a single program, strategy, or professional domain. To ensure that 
these outcomes arc achieved requires joint activity'. 

In short, outcomes can be used in three ways in community' reform efforts: 

• Creating a Commitment to Reform 

• Developing Strategies 

• Developing Accountability Systems to Produce Desired Results 

Defining Outcomes, Indicators, and Performance 
Measures 

If outcomes are to be used to guide reform there must be clarity in what is meant by 
"outcomes" and how "outcomes" are distuiguishcd from other measures. Different 
people have used different terms — "goals," "outcomes," "impacts," “results," 
"indicators," “benchmarks," “interim outcomes,” “process measures," "performance 
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measures/’ “impact measures/’ and “implementation measures” — to make these 
distinctions. In this guidebook, the terms “outcome/’ “indicator/’ and “performance 
measure” will be used. “Performance measure” further will be broken into “impact 
measure” and “implementation measure.” They have the following definitions: 

Outcome A condition of well-being for a child, family, or community that 
has broad, common understanding. 

e.g. healthy births: stable families: safe neighborhoods 

Indicator. A measure which helps quantify the achievement of the outcome. 

e.g. rate of low birthweight babies: child abuse and 
neglect rates: juvenile crime rates 

Performance Measure A measure of the intended effects of a program or 
strategy. 

Impact measure. A measure of the impact upon the child, family, or 
neighborhood served. 

eg. percent of pregnant teens served who stop smoking 
and practice good nutrition: changes in parental 
discipline practices at home: rates of recidivism of 
juveniles assigned to intensive supervision program 
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Implementation measure. A measure of the degree of 
implementation of a program or strategy. 

e.g. percent of teen parents who keep clinic 
appointments ; percentage of parents who complete parent 
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education curriculum; percent of juveniles who are 
involved in intensive supervision program activities 

The outcomes that communities select to address — healthy births, healthy children 
and adults, children ready for school, children succeeding in school, young people 
avoiding trouble, stable families, self-sufficient households, safe neighborhoods, 
communities with diverse and rich activities and social connectedness — should have 
a common understanding While a community may initially establish as a vision or goal 
that "all children succeed to their full potential ' 1 or that "families flourish in an environment 
that supports their aspirations ,' 1 such a vision or goal statement is subject to broad 
individual interpretation It requires more specificity to guide reform. Outcomes offer 
this level of specificity, within a common language that all can understand. At the 
same time, these outcomes arc not directly measurable by a single piece of data. 

Indicators are needed as quantifiable measures of the extent to which the outcomes 
are being achieved. Several different indicators are likely to be needed as measures 
of each outcome. These indicators should be capable of ongoing measurement and 
tracking over time. They measure some actual condition of well-being in children, 
families, or communities. They are not a measure of program, system, or strategy 
effort to affect that well-being. 

Performance measures are needed to assess the implementation and impact of specific 
programs or strategies. They directly measure the specific goals set out for each 
program or strategy. They may be quite different for different types of strategies, 
even when those strategies have the same desired outcomes in mind Importantly, 
however, if the strategies are consistent with the community's effort to improve 
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Apperpix 2-2: Overheads 
descnbmg different terminology 
nst?d to define outcomes, indicators, 
and performance measures 



outcomes, the strategies and their performance measures should be seen as helping to 
achieve the desired outcomes. Some performance measures for programs are impact 
measures , while others are implementation measures . A tutoring program for children 
may set as one impact measure "'improved student reading scores/’ Since its strategy 
is to work with parents so they help their children with reading at home, it may set as 
an implementation measure "'parent participation in tutoring sessions"’ and "'reported 
parent involvement in home reading activities/’ Both are valuable to collaborative 
reform efforts. While the impact measure may be the ultimate program goal, it cannot 
be achieved without accomplishing the implementation measure. 

Using these outcomes, indicators, and performance measures effectively to inform 
and guide reform efforts requires that community collaborates have a good 
understanding of and common language for these different levels of measurement. 

Establishing Outcomes 

Both the choice of outcomes and indicators and the process by which they arc selected 
are important to using outcomes to produce reform. 

First, it is important for the community to identify outcomes across dimensions of 
physical, social, psychological, educational, and economic well-being. While many 
collaborates begin with a specific outcome in mind — improving infant health, reducing 
adolescent parenting, curbing substance abuse, reducing school drop-out, or reducing 
juvenile crime — real gains on that particular outcome usually require joint actions 
across professional domains. It rarely is possible to produce a sustained change in one 
of these outcomes without also addressing and affecting others. Although funding 
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opportunities or media attention may focus initial attention on a particular outcome, it is 
important for community collaboratives to view that outcome as part of a larger set of 
outcomes the community seeks to achieve. 

Ultimately, the goal should be to recognize that these outcomes must be addressed as 
an interrelated whole rather than as separate and distinct. One of the reasons for 
establishing outcomes is to focus community attention on what it wants to achieve, 
rather than on the current programs and services it is providing. This should broaden 
thinking to identity and change the underlying conditions that are responsible for 
producing the outcomes. For instance, adolescent parenting is not simply an issue of 
adolescent sexual behavior. It relates to school performance and youths' sense of 
their future opportunities. Therefore, an effective strategy to reduce adolescent 
parenting likely involves educational and social as well as health activities. If successful, 
it contributes to improving school completion and to reducing juvenile crime as well as 
to reducing adolescent births. 

A number of communities have gone through the process of defining a constellation of 
outcomes and their indicators, as have such states as Oregon, Minnesota, Georgia, 
Iowa, and Washington. While these processes have produced some variations, there 
also is much similarity and common ground. Lisbeth Schorr has recommended “A 
Minimalist List'’ of outcomes for children and youth, shown in Table 2-2, that generally 
can be used as a starting point. Measures of all of these should be available on a 
community, and often on a subcommunity, basis. Appendix .2-3 provides several 
illustrations of outcomes that different communities and states have selected. 
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Table 2*2 

: CHILD AND YOUTH OUTCOMES 

A Core list to Serve as a Starting Point 

Healthy Births 

Lower rates of low birthweight births 

Higher rates of early prenatal care 

Lower rates of birth to single mothers under 1 8 

TwoYear-Olds Immunized 

Children Ready for School 

Immunizations complete . 

No untreated visions or hearing problems 
School-readiness traits as identified on sample basis 

Children Succeeding in Elementary, Middle, and High School 

As irxiicated by lower rates of school dropout, and by 

Arademic achievement measures demonstrating competency over challenging subject matter 
in grades 4, 8 and 1 2 

Children and Youth Health, Safe, Prepared for Productive Adulthood 

Children not abused or neglected 
Children living in own family 

Children living in families with incomes over the poverty line 
Youth avoiding 

Early unmarried childbearing 

Substance abuse 

Arrests for violent crime 

Suicide 

Homicide 

Accidental death 

Sexually transmitted diseases and HIV/ADS 
Young adults in school or employed 



Source: Schorr. Lisbeth. "The Case for Shifting to Results-Based Accountability." in Making a 
Difference: Moving to Outcome-Based Accountability for Comprehensive Setvice Reforms. Des 
Moines. IA: NCSI Resource Brief 7, 1 995. 
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The first issue discussed in establishing outcomes relates to substance-, what outcomes 
and indicators should be included and how they should be viewed. It is equally important, 
however, that the process by which outcomes are identified and established is an 
inclusive one The more that the process involves diverse constituencies that represent 
the community as a whole, the more it will produce the needed commitment to take 
action. 

While different constituencies are likely to have different perceptions of the causes of 
poor outcomes and the strategics needed to address them, they are likely to have more 
agreement over what they would like their children to achieve. It is for this reason that 
outcomes for children can frequently serve as a mobilization base within communities. 
Designing a process for effective participation by all groups in the community in 
establishing outcomes can help build community consensus and commitment. This 
process can and should involve surveys, focus groups, and other outreach efforts 
within traditionally under-represented groups in community decision-making. Some of 
these techniques are discussed in later Guidebooks on neighborhood organizing, public 
engagement, and parental involvement. 

One novel approach was taken in Hawaii, where leaders asked youth what they 
wanted Hawaii to look like in the future. The outcomes set out by Hawaii youth both 
helped involve youth and demonstrated that there is a vision, widely shared, among all 
residents of the state. 

Once outcomes are established, it is critical to develop indicators that can be used to 
chart community progress in achieving them over time. While the entire community 
should be represented in establishing outcomes, the development of indicators will 
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require more detailed and technical work. It is important to select or develop indicators 
that can be tracked over time. It is important they capture the meaning of the outcomes 
that have been established. There usually is value in selecting several indicators for 
each outcome. As much as possible, it is important to select indicators that can be 
examined on a neighborhood level and by important population subgroups. There are 
several useful reference works describing how to select outcomes and indicators, 
based on the strengths and limitations of different specific measures (see Appendix 2- 
5). 

Using Outcomes to Develop Strategies 

Once outcomes and indicators have been established, the next step is to develop 
strategies that can be used to improve them. This requires that the community 
collaborative understand what underlying conditions are contributing to the current 
outcomes and how those need to change to produce better ones. 

There has been substantial work, particularly in the prevention field, in describing 
those factors that place children at risk of experiencing such “rotten outcomes” as 
school failure, adolescent parenting, juvenile crime and delinquency, abuse and neglect, 
and substance use and abuse. Those factors that protect children and youth against 
experiencing “rotten outcomes” have sometimes been referred to as “resiliency factors” 
and sometimes as “protective factors.” Research has shown that this list of factors is 
quite similar, whedier examining the factors that “protect” against substance use and 
abuse, against adolescent parenting, against school drop-out, against juvenile delinquency, 
or against other “rotten outcomes.” 
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Tins docs not mean that some children, despite the presence of all these factors, do not 
fall victim to "rotten outcomes" or that some children, despite tire absence of many of 
these factors, do not excel. What it does mean is that these factors contribute heavily 
to the differences in outcomes experienced by children, and across subgroups and 
neighborhoods 

One formulation of the factors that contribute to high rates of success for children, 
across all dimensions of well-being, is shown in Table 2-3 below . 



Table 2-3 

Conditions Needed for Children to 
: Succeed at High levels 



■ Economic and physical security, within the home and neighborhood 

■ Environmental and public safety 



b A nurturing, stable family environment 
B Adult mentors and role models in the community 
■ Positive peer activities 

H Opportunities to exert effort and achieve success 



0 Health care for medical needs 
a Decent schools and schooling 

Q Access to professional services to treat any conditions or needs that may 
arise and that require professional care 



Source: Bruner, Charles. Realizing A Vision for Children, FamHes, and Neighborhoods Des 
Moines, I A: NCSI Working Paper. I 995. 
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In addition to a substantial base of evidence, the list in Table 2-3 makes common 
sense. For children and in neighborhoods where these factors are present, the rates of 
achievement of positive outcomes are great. For children and in neighborhoods where 
many protective factors are absent, children are at much higher risk of experiencing 
“rotten outcomes.” Some children may become passive and drop out; others may 
react against and strike out; and others may give up and tune out. Some may do all 
these and more. This speaks to the interconnectedness of the constellation of outcomes 
that a community develops. 



Seeking to define the underlying conditions or factors leading to undesirable outcomes 
(or the protective factors producing desirable ones) helps to move beyond linear 
approaches and beyond looking at categorical and professional service systems as the 
source for solutions. The current service systems largely have been constructed 
based upon such a linear approach. Educational needs are the responsibility of the 
classroom; health needs are the responsibility of the medical office; employment needs 
are the responsibility of the welfare office. 



Appendix 2-6: Annotated 
bibliography on conditions that lead 
to child success including Benard's 
synthesis of the resiliency literature 
and table adapted from Catalano 
and Hawkins - 

Appei xhx 2-7: Excerpt from B/uner, 
Beyond a Linear Mode! of Reform 



An analysis by a community collaborative of the factors needed to produce high rates 
of success on its outcomes helps look beyond the current categorical approach and 
take a more comprehensive view. This also represents an important step in moving 
beyond a “service” approach to a broader “community” approach. An important 
synthesis of the research literature on resiliency by Bonnie Benard stresses looking 
not just for risk factors but also for the resiliency factors that encourage children to 
make positive choices. The work of Catalano and Hawkins similarly shows the 
importance of a common set of protective factors to avoiding adolescent parenting, 
youth drug involvement, school failure, and juvenile delinquency. The conceptual shift 
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from a categorical and linear approach toward a holistic and comprehensive approach 
represents a major change in thinking for many community collaboratives. 

The description of these factors and conditions can help identify the changes needed 
to improve outcomes Communities can use this to conduct assessments or 
environmental scans, both on a community-wide basis and by subgroups and 
neighborhoods. In fact, in some parts of the community these conditions or factors 
may largely be in place for most children and families (and it is here one would expect 
to find achievement of outcomes at high rates). Still, there may be an identifiable 
group of children and families not well-connected to these conditions. Alternatively, 
one or more factors may be missing or in short supply. The assessment or environmental 
scan can help identify areas for action and change. 

Such an assessment or environmental scan requires a much broader focus than the 
traditional inventorying of public service systems. The work of John McKnight and 
John Krctzmann, which emphasizes an asset-based approach to community assessment, 
needs to be incorporated to help balance what is often an emphasis upon community 
deficits and problems with an emphasis upon community assets that can help to produce 
needed change 

Such a community assessment or environmental scan, whether done explicitly or 
implicitly, should identify areas where resiliency factors and support systems need to 
be bolstered. It should help establish a set of strategies that, taken together, can 
produce the needed set of factors or conditions.' In some instances, particularly for 
families and in neighborhoods where several factors are missing, multiple strategies 
will be needed to impact indicators and outcomes 




Appendix .2-6: Annotated references 
to asset mapping aixt community 
assessment guides 



22 



Guidebook 2 



One simple diagram of the types of strategies that may be needed to produce the 
factors and conditions that will produce high rates of success is shown in Table 2-4, 
below. 
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TABLE 2-4 





Positive 

Community 

Outcomes 



In this diagram, whole villages refer to the strategies to create, at the neighborhood 
level, the economic and social conditions described in the list of factors. The African 
proverb states, “It takes a whole village to raise a child.” This proverb suggests the 
need, particularly within disinvested neighborhoods, for approaches that help create 
“whole villages” or make “villages whole.” This includes insuring economic stability 
and opportunity. It also includes insuring that community networks of support — 
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including mentoring, sharing, and recognition — exist to meet the interests and abilities 
of village residents. 

Effective frontline approaches refer to strategies that assure that families who are 
socially isolated become connected to the community or village so they can provide 
the needed family support to nurture their children. In essence, collaboratives which 
seek to establish “family resource centers,” "home visiting services,” and other 
preventive services often are doing so based upon a belief (explicit or implicit), that 
there is a gap for many families in this area. 

Productive service systems refer to an array of public supports, including schools and 
professional services, designed to meet educational and other specialized needs of 
children and families. To be productive, these systems must be timely in response, 
sufficient to meet their specific objectives, efficient, coordinated with other systems, 
integrated into community life, and adaptive and learning 

One emphasis in the diagram in Figure 2-4 is on the linkages across the strategies. In 
particular, service systems must be responsive to effective frontline approaches and 
whole villages in order to be effective. They must contribute to and support the 
development of whole villages and not seek to substitute for them. 

By beginning with outcomes and tracing back to the conditions or factors needed to 
achieve them at high levels, community collaboratives can begin to establish new and 
effective strategics. 
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Developing an Accountability System to Produce 
Desired Results 

A third use of outcomes is in restructuring the manner in which programs, strategies, 
and systems are held accountable. This will require new systems of accountability 
both for any new strategies that are developed and for existing systems of services 
and supports. 

Individual strategies may be developed that are designed to produce particular factors 
or conditions that are needed to produce outcomes but that are not designed to produce 
all of them. These strategics should be held responsible for producing those factors or 
conditions. This involves developing performance measures, both implementation and 
impact, that relate to those factors. This may be particularly true for those strategies 
that are designed to strengthen community conditions or to engage individuals who 
have been socially isolated from sources of support. 

These performance measures for individual strategies need to be a part of ongoing 
assessment efforts in order to determine if individual strategies are being implemented 
and producing intended impacts. In some instances, the strategies themselves need to 
be developed, and feedback on how well they are doing in reaching their goals can 
contribute to important strategy refinements and enhancements. In other instances, 
this assessment can be used to determine whether they actually have contributed to 
producing the factors they were designed to produce and whether they deserve to be 
continued, expanded, or abandoned. 

Any strategy that is developed should be viewed as an integral part of an overall set of 
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strategics for achieving desired outcomes, with clear understanding of its particular 
role and responsibility. As an illustration of this approach to accountability, a six-level 
model has been proposed for use with comprehensive frontline family practice reforms. 
Similar approaches should be developed for each element of a reform strategy. 

Individual strategics should be held accountable to performance measures, but these 
do not always mirror the indicators selected to measure community outcomes. The 
overall combination of stratcgies^developcd or envisioned by the community 
collaborative, however, should seek to produce a change in those outcome indicators. 
The collaborative should seek to ensure that its members are collectively accountable 
for achieving gains on the overall set of outcomes and indicators. 

Three points in developing accountability systems for comprehensive reform efforts 
arc worth discussion. Again, Lisbeth Schorr has pointed to a number of potential risks 
in moving to results-based accountability: 

Funders, the public, and even program people, may underestimate how 
long it takes to achieve significant improvement in outcomes. 

Demands for documented results could drive programs away from 
achieving their mission of improving a broad range of results [and towards 
short-term gains on easier issues]. 

Funders could be led to confine their support to interventions whose 
effects are readily and quickly quantifiable. 

Some forms of outcome measurement could lead to labeling and 
stigmatizing children or families. 
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Agency accountability could be weakened as attention shifts to 
community-wide accountability efforts. 

The shift to results-based accountability could be seen as the solution 
itself rather than a means for making changes that can produce a 

solution. 

The shift to results-based accountability cannot be allowed to substitute 
for safeguards against fraud, abuse, poor services, inequities or 

discrimination. 

The first point about time deserves additional discussion. It is unrealistic to expect 
issues and problems stemming from generational causes to be addressed quickly, in 
election-year time spans. The desire for "quick results'’ can deflect attention away 
from tackling long-standing and serious conditions. Policy-makers understand that 
these issues are complex, but they also want accountability. Simple solutions may not 
solve complex problems, but policy-makers need to know that progress is being made. 

This will require a process of educating both policy-makers and the public. The more 
that the goals of individual strategies are made clear, they are connected to an overall 
strategy or theory of improving outcomes in the long-term that makes sense, and they 
are tracked and held accountable for accomplishing their specific goals, the more 
likely that the needed long-term commitment from policy-makers and the public can 
be secured. 

Second, it is important to stress that some strategies will be charting new territory in 
trying to create results that have proved elusive to existing programs and services. 
These innovative efforts should be evaluated and held accountable, but their value will 
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be in the lessons they leam as much as the successes they achieve. An accountability 
system must be established that rewards risk-taking and encourages modifications 
and refinements of strategies as risk-takers leam from experience. It is easy to fall 
into a trap of assuming that the solutions are already known, and it is merely a failure 
to implement effectively to produce results. In fact, succeeding is likely to require 
major innovations both in developing new services and strategies and in restructuring 
existing ones. Harold Williams has described the different approaches funders need 
to take when investing in innovation and assessing its impact (see Appendix 2-10). 

Third and finally, it is important to distinguish between demonstration efforts and 
comprehensive reforms. Demonstration efforts are not going to produce major changes 
in outcomes on a community-wide level, no matter how successful they are. 

This does not mean that demonstration efforts should not be employed. In many 
cases, the logical first step is to try a strategy on a demonstration basis. It is important 
to do so in a way that can inform larger efforts. This may involve careful targeting so 
the impacts can be examined on a neighborhood or subgroup level. 

Often, such reform efforts are not defined as focused demonstrations, but are over- 
sold as solutions. Unfortunately, even if such strategies are very successful with the 
families they serve, they usually are too small to produce a noticeable impact on 
indicators and outcomes on a community-wide basis. This also can lead to disillusionment 
among policy-makers and the larger public. 

It is critical to be up-front regarding the extent of the impact that implemented reforms 
can be expected to produce. Multi-million dollar "rotten outcomes'’ are unlikely to be 
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reversed by hundred thousand dollar reforms. No matter how successful an adolescent 
pregnancy and parenting program is in working with the fifty teens it serves, it will not 
produce a discernible community-wide impact on adolescent parenting in a community 
with two thousand births to adolescents a year. 
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In making use of outcomes to establish and guide reform efforts and to establish 
accountability for achieving results, community collaboratives should take the following 
steps. 

Step 1: Establish an inclusive process to define community 

outcomes 

I. / Involve key stakeholders to gain diverse perspectives and buy-in. 

Different parts of the community will have different perspectives on many issues 
confronting the community and its children, families, and neighborhoods. At the same 
time, there may be substantial common ground around underlying goals and outcomes 
that the community should commit to achieve. While the resulting outcomes may be 
very much the same, an inclusive process for establishing outcomes helps to build 
bridges across constituencies in the community and raise overall community 
commitment. A narrower process, where a few people set the outcomes, can build 
distrust and does not create the broad ownership needed to take concerted and prolonged 
action. 

In establishing outcomes, it is important to reach out for perspectives from those who 
may not otherwise feel invested in the effort. It is particularly important to gain the 
voices of those most likely to be distrustful of the value of this work. 
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Involving a diverse constituency in identifying and setting outcomes is the start of a 
trust-building process. Disinvested groups can give needed perspectives and help 
identify common ground. Nontraditional but important constituencies such as business 
leaders can be engaged and, through the process, more deeply appreciate the importance 
of addressing child and family issues. 

1.2 Define outcomes in terms of improvements in child (and family and 
community) well-being. 

There is a tendency for groups to seek consensus upon a variety of issues all at once, 
and, as a result, to confuse outcomes and strategies. For instance, a consensus- 
building process might identify the following as community issues: to reduce adolescent 
parenting and youth violence (outcomes), and to increase parental involvement in 
schools and increase the supply of child care (strategies). Increasing parental 
involvement in schools and the supply of child care may, in fact, contribute to achieving 
the outcomes the community desires. Making decisions about strategics, however, 
should be made after outcomes have been established. 

Generally, communities can reach consensus over the outcomes they desire for children. 
These should form the core of the constellation of outcomes a collaborative develops. 
It is a basic societal necessity to ensure that children grow into adults capable of 
sustaining that society. A comprehensive set of measures for child well-being also will 
require producing family and community well-being, although many collaboratives 
include family- and community-oriented outcomes on their lists, as well. 

1.3 Identify multiple outcomes of well-being and view them as interconnected. 
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Some collaborative efforts start with care outcomes in mind — reducing adolescent 
parenting, improving school completion, assuring children start school ready to learn. 
To achieve any of these outcomes, however, requires that children are safe and secure 
across other dimensions of well-being. A broad set of outcomes (across social, 
psychological, physical, economic, and educational dimensions) helps assure that 
communities identify and involve all relevant stakeholders and resources in developing 
effective strategics. 

Step 2: Develop measurable in dicators for th e selected outcomes 

2. 1 Ensure that indicators have credibility and can be tracked over time. 

It is important that measurable indicators be established for each of the selected 
outcomes. These indicators should be tracked over time and have credibility as 
accurately reflecting the outcome. The list of criteria for selecting outcomes and 
indicators developed by Kris Moore is shown in Table 2-5. 

2. 2 Establish a system for regularly tracking and reporting on them . 

In addition to selecting indicators, community collaboratives should establish a system 
for regularly tracking and reporting on them, at least on an annual basis for most 
indicators. Even though it may take substantial time for strategies to produce impacts 
on these indicators, it is important that a system of reporting be maintained, to keep 
attention focused on community goals. 
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1 . Comprehensive Coverage. Indicators should assess well-being across a broad 
array of outcomes, behaviors, and processes. 

2. Children of all ages. Age-appropriate indicators are needed at every age from 
birth through adolescence and covering the transition into adulthood. 

3. Clear and comprehensible. Indicators should be easily and readily understood 
by tlie public. 

4. Positive outcomes. Indicators should assess positive as well as negative aspects 
of well-beng. 

5. Depth, Breadth and Duration. Indicators are needed that assess dispersion 
across given measures of well-being, children's duration in a status, and cumulative 
risk factors experienced by children. 

6. Common interpretation. Indicators should have the same meaning in varied 
population sub-groups. 

7. Consistency over time. Indicators should have the same meaning across time. 

8. Forward-looking. Indicators should be collected now that anticipate tlie future 
and provide baseline data for subsequent trends. 

9. Rigorous methods. Coverage of the population or being monitored should be 
complete or very high, and data collection procedures should be rigorous and 
consistent over time. 

I 0. Geographically detailed. Indicators should be developed not only at the national 
level, but also at the state and local level. 

.1 I . Cost efficient. Although investments in data about U.S. children have been 
insufficient, strategies to expand and improve the data system need to be 
thoughtful, well-planned, and economically efficient. 

1 2. Reflective of social goals. Some indicators should allow us to track progress in 
meeting national, state and local goals for child well-being. 

13. Adjusted for demographic trends Indicators, or a sub-set of indicators, should 
be developed that control or adjust for changes in the composition of the 
pop ulat ion” over time which confound our ability to track well-being. Alternatively, 
indicators should be available for population sub-groups that are sufficiently narrow 
to permit conclusions within that sub-group. 



Source: Moore, Kristin. Catena for Indicators of Child Well-Being. Washington. DC: Child Trends. 1994 
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Step 3: Construct a “ theory "of what it takes to achieve community 

outcomes 

3. 1 Work outside the boundaries o f public service system programs and 

responses. 

One of the values of starting the discussion from the perspective of outcomes is that 
it is not bounded by existing services or professional responses. In fact, one exercise 
community collaboratives can undertake is to imagine two neighborhoods (and the 
families in them) — one where outcomes are achieved at high levels and one where 
they are not. Collaboratives can then describe how the two neighborhoods differ - 
— what exists in the neighborhood with high levels of success that is not present in 
the other neighborhood. It is likely that the differences are much greater than the 
presence or absence of a parent education program, a youth violence prevention 
effort, or school-linked services. The important point is to move beyond professional 
services as the solution to improving outcomes and to look comprehensively, including 
building upon informal or natural family and community support networks. This 
should lead to a list of factors or conditions similar to those shown previously in 
Table 2-3. 

3.2 Include relevant economic, social, educational, psychological, and 
community conditions. 

The examination of the factors or conditions should extend beyond the social, 
psychological, and educational dimensions with which community collaborations 
concerned with children and families may be .most familiar. It should also include 
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economic, housing, environmental, and public safety concerns. All these play roles 
in contributing to the well-being of children and families. 

Step 4: Design a set of strategies believed to be sufficient to 

produce these factors or conditions 

4. 1 Examine the needs of families who are socially isolated and vulnerable. 

Improving a constellation of outcomes on a community-wide basis is likely to require a 
number of different strategies. Some of these strategies will need to focus on specific 
children and families; others will require a focus on neighborhoods. 

As Table 2-4 suggests, one area of focus should be on those families with children 
who are socially isolated and vulnerable and not connected to community supports. 
These families may need support in setting goals, connecting with needed supports, 
and seeing their own opportunities and possibilities. This is an area where family 
support, care coordination, home visiting, or case management may be needed. Such 
frontline practices can help children, youth, and families connect with supportive villages. 
Strategies in working with these families should be developed to help these families 
build, for themselves and their children, some of the factors that are needed to achieve 
high levels of success. 

4. 2 Examine how needs may differ to reflect different neighborhood 
conditions or other subpopulation differences. 

As Table 2-4 also suggests, some of the factors leading to high levels of success 
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involve neighborhood conditions such as economic stability andjob opportunities, safe 
streets, the presence of adult mentors, a diversity of participatory activities for children 
and youth, and a high level of ‘"social capital' 1 or community activities that enforce 
norms of civic behavior. These may be present in many neighborhoods in a community, 
but they may require bolstenng in others. In distressed and disinvested neighborhoods, 
addressing economic opportunity may be critical to achieving success with children 
and families. Some subgroups in the population — such as persons with disabilities, 
families with limited English-speaking proficiency, or ethnic groups — face additional 
barners that must be faced by community collaboratives. 

Most of these factors or conditions cannot be created on a case-by-case basis with 
individual families or children. These protective or resiliency factors must be present 
within the community to a sufficient degree that children and families can become 
involved in ways that suit their individual needs. They cannot be ordered as discrete 
services. Since they are elements critical to success, however, communities must 
work to develop strategies that can produce them where they are in short supply. The 
implications for investments are large, but the benefits are correspondingly great. As 
Appendix 2-11 suggests, retum-on-in vestment modeling can help identify both the 
need for such investments and their potential long-term benefits. 

4 . 3 Examine how professional services and supports need to change . 

Frontline practices reaching out to socially isolated and vulnerable families and 
neighborhood-based strategies involved in “village building’’ generally represent new 
services and strategies. They address gaps not filled by current systems of public 
services and supports. 




Appendix 2-11: Excerpt trom 
Bmner, Where's the Beer? and 
annotated bibliography on return- 
on-investment approaches 
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Appendix 2-12: Excerpt from 
Bruner, A Framework for 
Developing and Holding 
Comprehensive Reform Efforts 
Accountable for Improving Child 
Outcomes 



In addition to these strategies, community collaboratives need to examine how existing, 
publicly-financed services need to change. In particular, these publicly-financed services 
must be connected with and complement other reforms in the community. Reaching 
out to socially isolated families and children is only half of the equation; schools and 
human service agencies must welcome those families and children and validate their 
efforts to grow and develop. 

These services must become connected with community-building efforts. The term 
‘‘seamless system” is overused, but it means more than effective hand-offs and referrals 
between professional agencies. It means a continuity in the way families and children 
are treated — with respect and as partners in the development process. It means 
being part of a seamless and consistent system of supports and expectations, from the 
perspective of children and families. 

In developing strategies, community collaboratives should look at existing services and 
see how they need to change to fit into the overall vision of a community where 
children succeed at high levels. As Appendix 2-12 illustrates, collaboratives then can 
develop ways to measure the contributions each system is designed to make. 

Step 5: Create accountability systems for individual strategies 

and for overall approach 

5. 1 Distinguish between internal and external uses of outcomes, indicators, 
and performance measures. 



As people often do not mean the same thing when they talk about outcomes, indicators. 
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and performance measures, they also mean different things when they use the term 
accountability. This is particularly true when people discuss outcome accountability. 
A useful distinction can be drawn between collecting and using indicators and 
performance measures internally (for evaluation and assessment to improve practice) 
and externally (to insure accountability through providing rewards or sanctions). Table 
2-6 shows this distinction: 

I Table 2-6 ?| 

IIJI Di$t5mcfe$ n the Use of information " 111 

Accountability (external use of outcomes) 

“Rewards and sanctions 
° Performance contracting 
“Results-based management 

Evaluation and Assessment (internal use of outcomes) 

0 Participatory/interactive evaluation 
■ Empowerment evaluation 
“Self-assessment 

“Continuous quality improvement 

It is important for community collaborates to use information in both ways. The 
internal use of information is essential to learning from the experience of putting new 
strategies into place. People implementing such strategies should not feel that such 
infonnation will be used against them. 



At the same time, it is important that people, programs, and strategies be held accountable 
for what they have agreed to do. This external accountability does not always need to 
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be tied to outcomes and their indicators, but it must be tied to some performance 
measures. 

Often, policy-makers and the public feel there is no real accountability (outcome or 
process) for public systems. Community collaboratives need to be clear on the external 
accountability measures they will use. They need to specify the attendant rewards, 
sanctions, and corrective action and adjustment measures that achieving or failing to 
achieve those measures will produce. 

5. 2 Use performance measures related to implementation for accountability 
over implementation. 

In the reform process, community collaboratives will need a new system for holding 
individuals and strategies accountable. This should be based upon the achievement of 
results (outcome-accountability) rather than the rote following of procedures (process- 
accountability). This does not mean, however, that every strategy should be judged on 
the basis of a specific long-term outcome indicator. There are several types of 
accountability that should be made a part of the overall reform effort. 

In many cases, communities will be charting new territory. Communities will be trying 
to develop services, supports, and opportunities that can create the factors that are 
needed for achieving their outcomes. For these innovations and any changes in practice, 
individuals and organizations should be held accountable for actually implementing the 
new strategy of practice. 

Before a new strategy can be evaluated for its impact, it must be implemented. This 
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' implementation accountability ’ essentially raises the questions, “Did you do what you 
said you were going to do?” and, if not, ''Did you encounter unexpected obstacles that 
prevented you from doing it and therefore made modifications to your effort?” Just 
because one has named a new strategy does not mean it has been implemented, 
particularly when the goal is to dramatically change the manner in which families or 
neighborhoods arc engaged and supported. 

This is the first level of accountability that must be established. It requires a clear and 
concrete enumeration of the strategy itself and development of performance measures 
to dctenninc whether it has been implemented as designed. Any rewards and sanctions 
applied to enforce accountability should be applied to individuals and organizations 
responsible for implementation. 

5. 3 Use performance measures related to impact for strategy development. 

The second level of accountability is in assessing whether, once implemented, strategies 
produce their intended impacts. This requires that a specific set of impact measures 
be established for each strategy. These may include some indicators developed to 
measure outcomes, but they also may be measures of the factors or conditions the 
strategy is designed to create. 

At this level of accountability, the strategy itself is under scrutiny. If it is successful in 
achieving its objectives, it should be retained and, where needed, expanded. Where it 
is unsuccessful, it should be modified or abandoned. This should lead to continuous 
improvement in strategy design and implementation. 
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5. 4 Use outcome measures for overall collaborative strategy refinement. 

The third level of accountability is for the overall set of strategies and changes and 
whether, taken together, they are producing the desired outcomes. Ultimately, this is 
the responsibility of the community as a whole, with the collaborative becoming die 
locus of this responsibility. 

The community collaborative can help build this sense of responsibility by regularly 
tracking and reporting on the outcomes and describing how the strategies are designed 
to impact them. This is a long-term process of public education that will be discussed 
at length in a later Guidebook. 

In addition, the collaborative should assess whether its strategies are effective in 
producing the intended impacts and, taken together, sufficient to produce impacts upon 
the community outcomes. Attention to the changes in community outcomes can lead 
to refinements of the overall strategy, modifications to the ‘theory” behind the strategy, 
and expansion of specific elements to fill needed gaps. 

Summary of Steps. Many of the steps presented here will be described in more depth 
in subsequent Guidebooks, particularly those related to reform strategies. Attention 
to outcomes should infuse the overall reform effort. This is the goal behind reforms. 
Table 2-7 summarizes the steps, and sub-steps, spelled out above. Table 2-8 provides 
a self-assessment checklist for community collaboratives in moving to results-based 
accountability. 
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Ilijlfve-Step 


Step 1 : 


Establish an inclusive process to define community outcomes 




1.1 


Involve key stakeholders to gain diverse perspectives and buy-in. 




1.2 


Define outcomes in terms of improvements in child (and family and neighborhood) 
well-being. 




1.3 


Identify multiple outcomes of well-being and view them as interconnected. 


Step 2: 


Develop measurable indicators for the selected outcomes 




2.1 


Ensure that measures have credibility and can be tracked over time. 




2.2 


Establish a system for regularly tracking and reporting on them. 


Step 3: 


Construct a "theory' of what it takes to achieve community outcomes 




3.1 


Work outside the boundaries of public service system programs and responses. 




3.2 


Include relevant economic, social, educational, psychological, and community 
conditions. 


Step 4: 


Design a set of strategies believed to be sufficient to produce these factors or 
conditions 




4.1 


Examine the needs of families who are socially isolated and vulnerable. 




4.2 


Examine how needs may differ to reflect different neighborhood conditions or 
other subpopulation differences. 




4.3 


Examine how professional services and supports need to change. 


Step 5: 


Create accountability systems for individual strategies and for overall approach 




5.1 


Distinguish between internal and external uses of outcomes, indicators, and 
performance measures. 




5.2 


Use performance measures related to implementation for accountability over 
implementation. 




5.3 


Use performance measures related to impact for strategy development. 




5.4 


Use outcome measures for overall collaborative strategy refinement. 
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Have you: 

□ Created a process for establishing outcomes that reflects community 
goals and gains commitment from diverse constituencies? 

□ Recognized the difference between outcomes, indicators, and 
performance measures and constructed specific outcomes and indicators 
that are capable of being measured over time? 

□ Identified the factors and conditions that collaborative members believe 
must exist to achieve the constellation of outcomes defined as 
important? 

□ Identified the changes that are needed to create these factors or 
conditions within the community? 

□ Created and implemented strategies that, taken together, can produce 
those changes? 

□ Established a system for holding individual strategies accountable for 
achieving their part in creating those conditions? 

□ Established a system for examining the overall set of strategies for 
success in creating the conditions and producing the overall 
constellation of desired outcomes? 
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Appendix 2=1 

Reports on the Well-Bemg of Chnldreini sumdl Yoimth 



About this Appendix : 

There are an increasing number of state and national 
reports that track the well-being of children and youth 
across a number of dimensions. Four national reports are 
described here; the first three also have state-level 
counterparts. 

Annotated Bibliography: 

Annie E. Casey Foundation. Kids Count Data Book: 
State Profiles of Child Well-Being 1997. Baltimore, MD: 
1997. 

Annually, the Annie E. Casey Foundation publishes 
a Kids Count Data Book that tracks a variety of 
indicators of child well-being over time, with state- 
by-state comparisons. Included as the ten KIDS 
COUNT state indicators in the 1997 volume were: 
percent low birth-weight babies; infant mortality rate; 
child death rate; rate of teen deaths by accident, 
homicide, and suicide; teen birth rate; juvenile violent 
crime arrest rate; percent of teens who are high school 
dropouts; percent of teens not attending school and 
not working; percent of children in poverty; and 
percent of families with children headed by a single 
parent. These indicators were selected because of 
their availability and comparability across states. The 
Foundation also funds projects in each of the fifty 
states to develop state-based data books, with state 
contacts noted in the back of the national book. 



Public Health Service. Healthy People 2000: 
National Health Promotion and Disease Prevention 
Objectives . Washington, DC: Department of Health and 
Human Services, 1990. 

This volume is the product of a national effort, 
involving 22 expert working groups and almost 300 
national organizations, to develop health goals for 
the nation. It compiles a great deal of statistical 
information establishes a broad range of age-related 
objectives and recommendations covering health 
promotion, health protection, and preventive services. 
Many states have developed similar documents to 
establish their own health-related objectives. 

National Education Goals Panel. The National 
Education Goals Report: Building a Nation of Learners 
1997. Washington, DC: U S. Government Printing Office, 
1997. 

This report provides both national and state data on 
twenty-six different indicators related to the eight 
educational goals: ready to learn; school completion; 
student achievement and citizenship; teacher 
education and professional development; 
mathematics and science; adult literacy and lifelong 
learning; safe, disciplined, and alcohol and drug-free 
schools; and parental participation. Published 
annually, the report provides both baseline data and 
update information and is also available on-line at 
http://www. negp. gov. 
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Federal Interagency Forum on Child and Family 
Statistics. America s Children: Key National Indicators 
of llv/l-Being Government Printing Office: 1997. 
This report provides national trend data on a number 
of indicators of child and family well-being, including: 
population and family characteristics, economic 
security, health, behavior and social environment, and 
education. Under Executive Order, there is now a 
requirement to the Federal Interagency Forum to 
produce an annual report on the most important 
indicators of child well-being. 



O 

ERIC 



Appendices 



Guidebook 2 



Appendix 2=2 

Defining Terms 

About this Appendix: 

People often use different terms to describe the 
difference between outcomes, indicators, impacts, and 
process measures. The first overhead, adapted from one 
developed by the Center for the Study of Social Policy, 
shows some of the terminology that has been used. 

Community collaboratives should select and use terms 
consistently, so people understand the issues that are 
being addressed. The second overhead, developed by 
the Child and Family Policy Center, provides illustrations. 



Source: 

Center for the Study of Social Policy, Washington, 
DC, nd. 
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!§§§| Discipline The Tower of Babble 'lllll 


Column A 


Column B 


Column C 


A condition of well-being for 


A measure for which data are 


A measure of effectiveness of 


children, families, or 


available, which helps quantify the 


agency or program service delivery 


communities 


achievement of (Column A) 




Choose one: 


Choose one: 


Choose one: 


1. Outcome 


1. Indicator 


1. Performance measure 


2. Result 


2. Benchmark 


a. Implementation measure 


3. Goal 


3. Milestone 


b. Impact measure 


4. Impact 


4. Other 


2. Program measure 


5. Other 




3. Other 
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Defitiitbn of Terms ■ ::! ^^^MlillliMlllB^W 


Outcome 


A condition of well-being for a child, family, or 
community that ha? broad and common understanding 
° healthy births; stable families; safe neighborhoods 


Indicator 


A measure which helps quantify the achievement of 
the outcome 

• lowbirthweight rate; child abuse and neglect rate; 
juvenile crime rate 


Impact Measure 


A measure of the intended impact of a program or 
strategy upon the child/family/neighborhood served 
6 Percent of pregnant teens served who stop 
smoking/practice good nutrition; changes in 
parental discipline practices at home; rates of 
recidivism of juveniles assigned to intensive 
supervision program 


Implementation Measure 


A measure of the degree of implementation of a 
program or strategy 

• Percent of teens who keep clinic appointments or 
participate in home visiting program; percent of 
parents who complete parent education curriculum; 
percent of juveniles who are involved in intensive 
supervision program activities 
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Appendix 2=3 

State and Community Examples of Outcomes and 
Indicators for Children and Families 



About this Appendix: 

A growing number of states and communities have 
gone through a process of defining outcomes and 
indicators. Some have focused exclusively upon child 
outcomes, others have included child and family outcomes 
and neighborhood outcomes, and some have focused 
upon a full range of outcomes. The following examples 
(from Rochester, NY; United Way of Central Iowa; Georgia; 
Missouri, and Oregon) are the results of inclusive, 
consensus-oriented planning processes. In each instance, 
the general outcome is first defined, followed by a list of 
indicators that measure that outcome. 

Examples: 

f : 

Healthy Births 

•Reduction in low birthweight babies 
•Early and continuous prenatal care 
•Fewer births to school-age females 

Children Ready for School 
•Completed i mmunizations 
•No uncorrected vision or hearing defects 
•No untreated health problems 
•Observable school readiness traits 



Children Succeeding in School 
•Regular attendance 
•Staying in school, fewer dropouts 
•Fewer suspensions 
•Academic achievement 
•Graduation to employment/post-secondary 
education 

Young People Avoiding Risk Behaviors 

•School-age pregnancy 
•Substance abuse 
•Juvenile delinquency 
•Arrests for juvenile crimes 

Family Stability 

•Safe and supportive living environment 
•School stability 
•Living in poverty 
•Employment 

•Reduced child abuse/neglect 
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Healthy births evidenced by lower rates of: 

•Low birthweight babies 
•Late or no prenatal care 
•Births to school-age females 
•Newborns in intensive care 
•Infant mortality 

Positive, Safe & Caring Environment for Early 
Childhood evidenced by: 

•Completed immunizations for age two 
•No preventable or untreated health problems 
•Living in own family or stable foster care 
•Health insurance coverage 
•Number of children enrolled in Head Start or 
pre-school, early care and education 

Children Succeeding in School evidenced by: 

•Academic achievement measures 
•Attendance/lack of truancy 
•No placement in special education 
•No retention in grade 
•No suspensions 

•Not being a victim of sexual or child abuse 

Positive Youth Development evidenced by avoiding: 

•School age pregnancy 
•Substance abuse 
•Juvenile arrests 

•Diagnosed cases of HIV/AIDS (0-19 years) 
•Homicide (0- 1 8 years) 
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Stable Families evidenced by : 

•Percentage above poverty level 
•Low mobility 

•Decreased domestic violence 
•Divorce rates 

•Quality, affordable housing 



for /: 

Children and Families 



Healthy Children 

•Increase the percentage of babies born healthy 
(weighing 5.5 pounds or more, born to mothers 
who received prenatal care the first trimester, 
and born to mothers who did not smoke or drink 
alcohol during pregnancy) 

•Increase the percentage of children 
appropriately immunized by the age of two 
•Reduce the pregnancy rate among school-age 
girls 

•Reduce the percentage of children who have 
untreated vision, hearing, or health problems at 
school entry 

•Reduce the teenage homicide rate 
•Increase the percentage of youths who do not 
use alcohol, tobacco, or illegal drugs 

Children Ready for School 

•Increase the percentage of low-income students 
in Head Start or prekindergarten programs 
•Increase the percentage of kindergarten students 
who attended preschool or child care programs 
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® Increase the percentage of students passing the 
Georgia Kindergarten Assessment program 
°Reduce the percentage of students who are two 
or more years overage in the third grade 

Children Succeeding in School 

“Reduce the percentage of students who are 
absent ten or more days from school annually 
“Increase the percentage of students performing 
above state standards on curriculum-based tests 
at the fifth and eleventh grades 
“Increase the percentage of students scoring 
above the national median on normed 
achievement tests at the eighth grade 
•Increase the percentage of students who 
graduate from high school on time 
•Increase parental involvement 

Strong Families 

•Increase the percentage of stable new families 
(with the first birth to a mother who has completed 
high school and is age 20 or older, and with the 
father ’s name recorded on the birth certificate) 
•Reduce the percentage of teenage mothers who 
have a second or higher-order birth before the 
age of 20 

•Reduce the incidence of confirmed child abuse 
or neglect 

•Increase the percentage of children in foster care 
who are placed in a permanent home 
•Reduce the percentage of youths arrested 




Self-Sufficient Families 

“Reduce the percentage of children living in 
poverty 

“Reduce the percentage of female-headed families 
living in poverty 

“Increase the percentage of welfare recipients 
leaving public assistance because of 
employment or higher incomes 
“Increase the rate of growth in employment 
“Reduce the unemployment rate 
“Increase affordable, accessible, quality childcare 




Healthy Child and Youth Development (Children 
Ready to Enter and Succeed in School) 

•Rate of infant and youth mortality 
•Rate of infant low birthweight 
•Immunization rate of 2-year-olds 
•Rates of sexually transmitted diseases 
•Alcohol and drug abuse rates 
•Expanded access to developmentally 
appropriate, early care and education 
•Incidence of clinical depression 
•Rate of “learning readiness” at school entry 
•Average educational achievement for age 
“Graduation rates of 7th and 9th graders 
•Indicence of fetal alcohol and drug trauma 

Strong Neighborhoods and Families (Children and 
Families Safe and Healthy) 

•Rate of parental child abuse and neglect 
•Rate of teen pregnancy 



°Rate of out-of-home placement 
•Rates of delinquency and habitual delinquency 
•Rate of child support default 
•Increased resident involvement in 
neighborhood-based collective action 
•Rate of violent crimes 
•Rate of spousal abuse 

•Average number of residents per housing unit 
•Percentage of income paid for rent 
•Income above poverty 

•Increased “success" of neighborhood collective 
action efforts 

Parents Working 

•Percentage of high school graduates with BA 
or higher 

•Community unemployment rate 
•High school graduation rate 
•Jobs in new or expanded community-based 
businesses 

•Percentage of high school graduates with other 
post-secondary degrees 
•Employment-to-population ratio 
•Percentage of high school graduates employed 
within 90 days 

•Accessibility of transportation from community 
to work and child care sites 




Nurture Children, Strengthen Families 

•Reduce teen pregnancy rates 
•Improve early childhood development 
•Reduce teen drug use 
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Improve Public Safety 
•Reduce juvenile crime 

•Increase the number of communities involved 
in community-based law enforcement planning 

Give High School Graduates the Essential Skills 
Needed for Success in Life 

•Increase the number of high school students 
who meet the standards for a certificate of 
initiative mastery 

Leave No One Behind in Oregon Life 

•Reduce the percentage of Oregonians who live 
in poverty 

•Increase the percentage of our high school 
graduates going on to college 
•Maintain or increase the share of employment 
among Oregonians who live outside the 
Willamette Valley 

Increase Health Care Access, Effectiveness 

•Improve the economic access of Oregonians to 
health care 

•Stabilize and reduce HIV cases 

Manage Community Livability 
•Improve air quality 
•Reduce housing costs 

Protect Natural Resources 
•Increase wild salmon runs 
•Protect air quality 

Improve Public Service Delivery 

•Increase agencies who use performance 
measures 
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Appendix 2=4 

Yountlhi Visions — Hawaii’s Approach 



About this Appendix: 

In Hawaii, leaders asked youth to develop their own 
vision for Hawaii's future, with a Youth Steering Committee 
compiling the results of an islands-wide survey of youth. 
Supported by the Hawai i Communities Services Council 
and the Hawai'i Community Foundation, the result is being 
used to guide the state's benchmarking process. This 
approach helped bring youth into focus as a resource and 
asset rather than a problem. 

For More Information: 

Hawai i Community Services Council, 200 North 
Vineyard Blvd., Suite 4 1 5, Honolulu, Hawai'i 968 17 (tel. :808- 
539-3560; fax: 808-539-3555). 

I Vision Statement and Commentary: 

Ke Alla Hoku 
The Children’s Vision 

The ENVIRONMENT of Hawai'i is clean and pollution- 
free, where native plants and animals are abundant, land 
is preserved, and recycling is a common practice. 

SAFETY is assured in our communities due to the absence 
of violence, gangs, and weapons. 

Our ECONOMY includes lowered taxes and cost of living, 
and affordable housing. It is multi-based, with many job 
opportunities for teenagers and adults. 



The DEVELOPMENT of hotels, golf courses, and 
infrastructure has occurred only with necessity and with 
the preservation of the land. 

EDUCATION in Hawai'i is provided through an ample 
amount of schools, with competent and qualified faculty. 
Hawaiian culture courses are widely available. 

The PRESERVATION OF THE ALOHA SPIRIT has kept 
the unique culture ofHawai'i alive. 

The SOCIETY ofHawai'i thrives on equality and respect 
for all. The attitude has led to better government, the end 
of homelessness and racism, and the control of over- 
population. 

DRUGS in Hawai'i are taken in a legal non-excessive 
manner. 

NATIVE HAWAIIANS have a strong voice in government 
and have been compensated for losses. 

Public TRANSPORTATION is provided to everyone, 
reducing traffic problems. 

RECREATION for youth has been provided through 
abundant facilities and activities. 

The HEALTH of the people of Hawai'i has improved 
through prevention of diseases. 



More modern TECHNOLOGY, new inventions and 
alternative energy are readily available and frequently used 
by everyone. 



— We believe we can make a difference, 
especially when we work with other groups and 
individuals towards a common goal. We believe 
we can turn our vision into reality. We believe in 
leadership combining youth and adults. We 
believe youth must take action to improve their 
future lives. 

— Members of the Ke Ala Hoku Youth Steering 
Committee, 1996-1997 
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Resources for Sefiectirng Indicators off Child WeEI-Beiimg 



About this Appendix: 

There are several works that are of particular use in 
selecting indicators of child well-being. Three of these 
are referenced below, with brief annotation. 

Annotated Bibliography: 

Improved Outcomes for Children Project. Finding 
the Data: A Start-Up List of Outcomes Measures with 
Annotations. Washington, DC, Center for the Study of 
Social Policy, 1995. 

For each of the indicators shown in Table 2-2, 
this document provides the following: a definition, 
a statement of its significance, facts about its 
prevalence in society, data sources from which it 
may be collected, commentary about its amenable 
to subpopulation analysis and comparability 
across states and over time, and other measures 
related to it. The indicators described are: low 
birthweight, prenatal care, non-marital teen births, 
immunizations, untreated vision or hearing 
problems, abuse and neglect, children in out-of- 
home care, child poverty, high school dropout, 
substance abuse, violent youth crime, teen 
suicide, teen homicide, accidental deaths, 
sexually transmitted disease and AIDS and HIV 
among teens, and youth idleness. 



Love, John, ‘'Can We Measure the Results?,” in 
Kagan, Sharon Lynn, Sharon Rosenkoetter, and Nancy 
Cohen (eds.) Considering Child-Based Results for Young 
Children: Definitions , Desirability, Feasibility, and Next 
Steps. New Haven, CT: Yale Bush Center in Child 
Development and Social Policy, 1997. pp. 48-71. 

This paper describes current state-of-the-art 
methods to measure child results, with particular 
attention to child results in the context of the 
school readiness assessment. It includes a 
discussion of the issues involved in measuring 
developmental progress at early ages, and 
includes a number of possible measures, their 
sensitivity to community interventions, their 
strengths and liabilities as a measure, and their 
policy relevance. 

Moore, Kristin. Criteria for Indicators of Child 
Well-being. Paper prepared for Conference on “Indicators 
for Children’s Well-Being.” Washington, DC: Child Trends, 
Inc., 1994. 

This paper discusses thirteen different criteria, 
shown in Table 2-5, that should be considered in 
selecting indicators of child well-being for use in 
community or state planning. 
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Appendix 2-6 

Copditioims That Lead to Child Success 



About this Appendix: 

There is a great deal of literature describing the 
correlates for child success and failure. The following 
annotated bibliography represents good syntheses of this 
literature. The chart, adapted from the work of Catalano 
and Hawkins, shows that adolescent problem behaviors 
are the consequence of a common "pool" of risk factors. 

Annotated Bibliography : 

Benard. Bonnie. Fostering Resiliency in Kids: 
Protective Factors in the Family ; School and Community. 
Seattle. WA: Far West Laboratory. 1994. 

This monograph provides an excellent synthesis 
of the literature, primarily from the field of 
"prevention," that describes the different 
protective factors — traits, conditions, situations, 
and episodes — that enable youth to overcome 
stresses and challenges. Benard first offers a 
profile of the resilient child and then describes 
the protective factors within the family, school, 
and community that foster that resiliency. 

Haveman, Robert and Barbara Wolfe. 
Succeeding Generations: On the Effects of Investments in 
Children (Russell Sage Foundation. New York, NY: 1994). 

This book draws from two decades of longitudinal 
data to evaluate the impact of many background 
factors — including parent education, family 
structure, and neighborhood environment — to 
determine which are most strongly associated 



with child educational success, adolescent 
parenting, and early adult employment. It 
concludes that parental educational status and 
neighborhood environment all are strongly 
correlated with child educational, social, and 
economic success. 

Schorr, Lisbeth (with Daniel Schorr). Within Our 
Reach: Breaking the Cycle of Disadvantage . New York, 
NY: Anchor Press. 1988. 

The first two chapters of this book. "The High 
Cost of Rotten Outcomes" and "The Risk 
Factors," describe the heightened likelihood of 
"rotten outcomes" (adolescent violent crime, 
leaving school uneducated, school-age 
childbearing) in situations, particularly those 
characterized by persistent poverty, where risk 
factors concentrate. 
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f5: Adolescent Problem Behaviors 


Risk Factor 


Substance 


Delinquency 


Teenage 


School 


Violence 




Abuse 




Pregnancy 


Dropout 
















Community Laws and Norms Favorable Toward Drug Use, Firearms, and 
Crime 




✓ I 






y 


Transitions and Mobility 


✓ 


/ 




y 




Low Neighborhood Attachment and Community Organization 


✓ 








y 


Extreme Economic Deprivation 


✓ 


✓ 


y 


y 


y 




; v ' " " \ ’ y i 


: 


"'VKSF jijiii 






Family History of the Problem Behavior 


y 


/ 


y 


y 




Family Management Problems 


y 


/ 


y 


y 


y 


Family Conflict 


✓ 


1 y 


y 


y 


y 


Favorable Parental Attitudes and Involvement in the Problem Behavior 


y 


y 






y 




! * ] ! 










Early and Persistent Antisocial Behavior 


✓ 


✓ 


✓ 


✓ 


y 


Academic Failure Beginning in Elementary School 


y 


/ 


y 


y 


y 


Lack of Commitment to School 


y 


/ 


y 


y 
















Rebelliousness 


/ 


✓ 




y 




Friends Who Engage in the Problem Behavior 


✓ 


/ 


/ 


✓ 


y 


Early Initiation of the Problem Behavior 


S 


/ 


y 


y 




Constitutional Factors 


✓ 


/ 






y 


Academic Failure Beginning in Elementary School 


✓ 


/ 


y 


✓ 


y 


Lack of Commitment to School 


2 


/ 


y 


y 








■| : 


!!! 






Rebelliousness 


S 


✓ 




y 




Friends Who Engage in the Problem Behavior 


2 


y 


y 


y 


y 


Favorable Attitudes Toward the Problem Behavior 


/ 


y 


y 


y 




Early Initiation of the Problem Behavior 


7 


y 


•/ 


7 


y 


Constitutional Factors 


2 


y 






y 



Source: Catalano and Hawkins , Risk Focused Prevention, Using the Social Development Strategy ; 1995 
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Appendix 2=7 

Beyond a Linear Model to Reform 

About this Appendix: 



A common critique of the current system of services 
and supports is that it is categorical rather than holistic, 
with individual services designed to address only one 
service need (health, education, safety, etc.). The excerpt 
argues that, in moving to outcome-accountability, people 
should not make the same mistake in considering outcomes 
as separate and distinct from one another. 

Source : 

Bruner, Charles. Beyond a Collaborative Model: 
Moving to a Holistic Approach to Systems Reform. Des 
Moines, IA: Child and Family Policy Center Occasional 
Paper, 1996. 

Excerpt: 

Community collaborates often start because a group 
of individuals recognize they need to work together to 
address the needs of children. Children bring more than 
educational needs into the classroom; pregnant women 
bring more than medical needs into the health clinic; 
families bring more than employment needs into the welfare 
office. Even if the primary goal is to improve educational 
performance, assure infant health, or establish family 
economic self-sufficiency, these other needs often must 
be addressed to make progress on the primary goal. 

In essence, this is a rejection of a simple, linear (or 
social engineering) model for improving the status of 
children — shown below for education. 



Figure One: 

Linear Model to Educational Success 











Educational 


Child 


<=> 


School 


<=C> 


Mastery 


Graduation 
Test Scores 











This linear model doesn’t work for many children 
because it doesn’t involve the family, at the left hand side, 
and because it doesn’t address non-educational needs 
and concerns, in the center. 

Many community collaboratives begin with a second, 
collaborative model, shown below, to improve a specific 
outcome such as school completion. 

Figure Two 



Collaborative fYiodel to Improved - Optimal 
Child Educational Development 
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This model includes the family as well as the child, 
and establishes linkages to address a variety of non- 
educational needs the child and family may have. It still, 
however, places education — both as goal and as manager 
of the process — at the center of reform. 

By changing these labels, of course, the collaborative 
model also serves to address other specific concerns, 
shown for child health below. 

Figure Three 



CoMaboralUvo IStodteD to 
Improved - Optimal ©(hold Health Statws 




While the same players and systems are included in 
this model, the health care system and health goals are 
now at the strategic center. In fact, a number of different 
collaboratives often have developed within a community, 
based upon this collaborative model. While each may 
have a different end-goal — reducing adolescent 
pregnancy, reducing teen violence, improving school 
readiness, reducing infant mortality, reducing child abuse 
and neglect, increasing school completion — they usually 






share an orientation — as well as the need for similar 
partners in collaboration — to achieving that goal. They 
also often have identified the same families and children 
for their services. This is what has sometimes been referred 
to as a “fragmentation of collaboration” within 
communities. 

Some communities are beginning to seek to establish 
a third model — a comprehensive and holistic model — 
that sets as a goal the well-being of children across a 
variety of dimensions of well-being. This model recognizes 
the need to develop a system of strategies, services, 
supports, and opportunities that work together to assure 
the full range of child outcomes. While communities still 
may have a variety of individual collaboratives focusing 
upon specific concerns, these must be linked to 
complement one another in a broader overall scheme. This 
comprehensive and integrated model is shown below: 

Figure Four 

Commuim'ty Goads for 
Overall Systems of Care 



Optimal - 
Satisfactory 




This is a departure from the collaborative model in 



two important respects. 
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First, as the model shows, community collaboratives 
with a comprehensive focus must develop a series of 
outcome measures that accurately capture the dimensions 
of child well-being they seek. These outcomes must be 
seen as an interrelated constellation of outcomes and not 
as discrete and independent occurrences. It is not enough 
to achieve success on one or two dimensions of well- 
being. Sustained gains are possible only if they occur 
across all dimensions. 

Second, the systems of support that are needed are 
not necessarily the purview of one, identified agency or 
professional discipline. Strategies are required that take a 
family-focused, consumer-driven approach, which requires 
responses from different agencies or professional 
disciplines at different times to address specific family 
needs and opportunities. While different systems 
(education, health, mental health, human services, 
community-based family support, etc.) may take the lead 
with specific families, the overall system must extend to all 
families. This frontline work (sometimes referred to as 
case management or care coordination, sometimes as family 
development or family support) must be able to secure 
needed responses from all agencies and professional 
domains as the child or family needs them. 

In essence, this holistic, frontline work with families 
is at the strategic center, with all agencies and professional 
domains responsible to it. While existing agencies and 
professional domains may provide this frontline work for 
certain families, it is the community’s broader responsibility 
to ensure this frontline work is available to all families in 
settings that they select and are more appropriate to their 
needs. 
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Appemidox 2-8 

Mapping Commmiiniilty Strengths 



About this Appendix: 

New tools and strategies have been developed to 
look at neighborhoods and communities from a strengths 
- as opposed to needs - basis. John McKnight and John 
Kretzmann. in particular, have developed conceptual and 
practical tools to map community assets and strengths. 
As McKnight and Kretzmann stress, many community 
assessments inventory neighborhood needs or deficits, 
but community-building must build from strengths and 
assets, not deficits. The Family Resource Coalition has 
produced a practical guide. Know Your Community , that 
provides a comprehensive approach to community 
assessment that incorporates asset mapping. The United 
Way of Cincinnati has used McKnight and Kretzmann’s 
asset mapping approach and described its impact upon 
their work. This appendix briefly describes some of the 
literature on asset mapping. 

Annotated Bibliography: 

McKnight, John, and John Kretzmann. Mapping 
Community Capacity . Report of the Neighborhood 
Innovations Network. Chicago, IL: Center for Urban Affairs 
and Policy Research, Northwestern University. 1991. 

This article provides an excellent, brief conceptual 
overview of the difference between needs 
mapping and asset mapping and the power and 
potential for taking an asset approach to 
community assessment. 



McKnight. John and John Kretzmann. Building 
Communities from the Inside Out: A Path Toward Finding 
and Mobilizing a Community's Assets. Evanston, IL: 
Center for Urban Affairs and Policy Research. 
Northwestern University, 1993. 

This book provides the nuts-and-bolts for making 
contact with a wide array of community-based 
organizations and affiliations in mobilizing a 
community for change. It senes as a resource 
and guide for identifying community assets and 
resources, including discussions of the different 
assets and resources that exist in communities 
and how to approach and engage them. 

Samuels, Bryan, Nilofer Ahsan, and Jill Garcia. Know 
Your Community: A Step-by-Step Guide to Community 
Needs and Resources Assessment. Chicago, IL: Family 
Resource Coalition, 1996. 

This guide describes a five-part process to 
community assessment. (1) establishing a 
community planning team, (2) defining community 
boundaries. (3) developing a statistical profile, (4) 
assessing needs from residents’ perspectives, and 
(5) identifying assets and resources. It includes 
many tools for collecting information, and 
emphasizes resident involvement in all aspects of 
community assessment. 
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United Way and Community Chest of Greater 
Cincinnati. Report on Community Capacity Building. 
Unpublished document, n.d. 

The United Way and Community Chest of Greater 
Cincinnati worked with John McKnight and his 
colleagues in conducting asset mapping in several 
Cincinnati neighborhoods, as part of their 
community capacity building efforts. This report 
describes the work in Cincinnati, including 
lessons learned and challenges and opportunities 
to implementing an asset mapping process as 
part of a larger community capacity building 
initiative. 
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Appendix 2-9 

A Framework for Measuring Comprehensive Service Reform Strategies 



About this Appendix: 

Ultimately, the goal of most community collaboratives 
is to produce change in the child and family well-being on 
a community-wide level. Specific service reforms, however, 
rarely can be expected to have such an impact without 
other changes at the community level. Many times, 
collaboratives develop new, more preventive and 
comprehensive service approaches. “A Framework for 
Measuring the Potential of Comprehensive Service 
Strategies" poses a six-level approach for evaluating such 
approaches. Only when the first five levels have been 
achieved, the article argues, can impacts be expected on a 
community-wide basis. The Table, drawn from this article, 
raises questions that need to be addressed at each of the 
levels. 

Source: 

Bruner, Charles, “A Framework for Measuring the 
Potential of Comprehensive Service Strategies,” in: Young, 
Nancy. Sid Gardner, Soraya Coley, Lisbeth Schorr, and 
Charles Bruner. Making a Difference: Moving to Outcome- 
Based A ccountability for Comprehensive Sen’ice Reforms . 
Des Moines. IA: National Center for Service Integration, 
1994. pp.2940. 
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Assessmertt/EvaluatHM 

A. Service Penetration 

•What are the characteristics of families most 
likely to benefit from the comprehensive service 
strategy being developed? 

•How do these correspond to the families who 
are receiving services and supports through that 
strategy? 

•What proportion of families within the 
community with these characteristics are being 
reached by the service strategy? 

B. Family Engagement 

•To what extent do families initially reached by 
the service strategy “join” and begin 
participation? 

•How long do they continue to participate? 
•What are their reasons for ceasing to 
participate? 

C. Family Growth 

•What growth occurs for families during their 
time of participation? 

•To what extent do families attribute growth and 
achievement of goals to some support provided 
by the service strategy? 
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•For what proportion of the families do the 
workers feel needed change has occurred, and 
in what ways has the service contributed to that 
change? 

D>. Community Embeddedness 

•What visibility do the services/strategies have 
within the community? 

•What proportion of families in the community 
are aware of the services and. whether or not 
they participate, what is their orientation to them? 
•How do other community institutions and 
networks of community support view the 
services, and to what extent do they see them 
fulfilling a needed community role? 

•To what extent have the services/strategies 
helped community residents define community 
need and set community goals? 

E. System Response, Climate for Reform, and Change 
•To what extent have the providers of 
comprehensive services identified and 
advocated for changes and new responses 
within other public service systems? 

•What changes have (and have not) occurred 
within other service systems that may be 
attributed to this advocacy? 

•To what extent have community residents 
identified and advocated for changes and new 
responses from other public service systems 
and from society as a whole? 

•What changes have (and have not) occurred 
within other service systems and within society 
as a result of this advocacy? 
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F. Community-Wide Family Well-Being 

•What changes in indicators of family well-being 
on a community-wide (both within and across 
neighborhoods within a community) basis have 
occurred since the initiation of new service 
strategies? 

•Given the penetration of the service strategy 
and the extent to which other services have been 
able to respond to identified family needs, how 
much impact could the strategy realistically be 
expected to make? 

•To what extent do community residents attribute 
any changes in the quality of life within the 
neighborhood and community to the changes 
in services? 
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Appendix 2-10 

Innovation and Outcome-Accountability 



About this Appendix : 

This excerpt is from Chapter 17, “Buying Programs: 
Buying Change,” in Harold Williams' Outcome Funding. 
The overall book presents a different approach to 
grantmaking and budgeting than traditionally found in 
government- and foundation-funded projects and is useful 
reading for those seeking to produce change through 
financing new programs or projects. 

Chapter 17 discusses the importance of investing in 
innovation, with investors needing to recognize that 
failures as well as successes can yield valuable dividends 
in moving toward more effective services. From their 
experiences with a wide array of innovative investments, 
the Innovation Group at The Rensselaerville Institute has 
developed twelve guidelines for making investments in 
innovation. 

Source: 

William, Harold. Outcome Funding: A New Appmach 
to Targeted Grantmaking , 2nd Edition . Rensselaerville, 
NY: Rensselaerville Institute, 1993. [Excerpts taken from 
pages 194-216] 

Excerpt: 

An innovation is best described as a response to 
these three questions: 

1. What s wrong? What is it about existing 
programs and products that is proving 
ineffective? Alternatively, what is the gap or 
niche that existing products do not serve? 
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2. What s new? In terms of either a new premise 
or refinement, what do you propose to do that 
will eliminate or at least substantially relieve the 
condition you have described above as 
problematic? Please be very specific about the 
key features of your approach. 

3. What s the project? What specific project will 
you undertake that will explicitly test the ability 
of your innovation to outperform present 
practice? Be highly specific about the results 
that you forecast, how they are different from 
the baseline of performance now achieved, and 
the products your innovation would improve or 
replace. 

Many funders mistakenly believe that the desired 
outcome from innovative projects is the same as for other 
projects, an immediate gain in program services. While 
this may occur, it is a secondary benefit. The primary gain 
is the testing of a new approach that shows promise of 
outperforming a present practice. In this context, as much 
gain can accrue from a failure as from a success. In our 
experience with hundreds of innovations, we have found 
that the most frequent outcome of a project that does not 
work is that other innovations are suggested which offer 
an increased likelihood of success. 

The key to capturing the learning and change value 
of innovations is investor commitment to do something 
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with innovation findings in hand. This is a critical matter/ 
since the incidence of effective spread of innovations is 
probably rarer than innovations themselves. The 
proportion of pilots, models, and demonstration projects 
that are never replicated at all is discouragingly high. 

Two investor steps beyond project financing are 
critical. The first is to capture the informational value of 
the innovation project. Project analysis must address the 
question. “So what?" For those innovations that work on 
a small scale, what changes in larger policies and programs 
are suggested? To what activities and enterprises are 
both the findings and the innovation itself relevant? 

The second step of applying learnings from an 
innovation must also be considered. Without initiative, 
innovation findings are not likely to be used, no matter 
how clear and compelling they may be. The reasons for 
this come from a most useful body of research on diffusion 
of innovation conducted by Everett Rogers and colleagues 
(Diffusion of Innovation. New York: The Free Press, 1982). 
The research has two key findings. First, innovations are 
much more effectively transmitted by a personal agent 
than by impersonal means, such as books and the 
electronic media. Second, most people are simply not 
ready to try something new, even when someone else has 
proven that it works. The predisposition of humans to 
use an innovation begins with a very few “early adopters." 
Then come the “mainstreamers," who will embrace a new 
practice only when it has been shown to work by an early 
critical mass. Until that happens, no amount of information 
or documentation is likely to lead them in that direction. 
Investors can find early adopters by shaping specific 
investment rounds which invite them to step forth and to 
apply an innovation. This strategy can be effectively 
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combined with the personal agent theme. 

We close with 12 guidelines to investors in 
innovations. They are based on tested assumptions for 
innovation. 

1 . Be willing to place your bets on individuals. 
Innovation does not come from books, budgets, 
or plans. It comes from individuals who have a 
vision for a better way and the project to express 
it. Innovation is rarely pre-formed in a tidy and 
complete program description. Its potential is 
better gauged by the presence of one or more 
people who are highly knowledgeable about 
alternatives to the status quo and who have the 
persistence to reach their goal. 

2. Be prepared to invest in some unruly people. 

Innovators may not make ideal supplicants. 
They can be irritatingly self-assured. Their need 
for achievement may well dominate their need 
for affiliation, and they may gain their energy by 
having been fired or rejected by someone else. 
Remember, you are choosing an innovator, not 
an employee. The same traits that would lead to 
prompt rejection for hiring may prove essential 
in sustaining a difficult innovation. Remember 
George Bernard Shaw’s comment: “The 

reasonable man adapts himself to the world: the 
unreasonable one persists in trying to adapt the 
world to himself. Therefore, all progress depends 
on the unreasonable man." 

3. Look for entrepreneurial characteristics. Ask 
would-be innovators about past failures and 
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listen carefully. Do they blame their failures on 
external forces, or look squarely at their own 
mistakes? Did they learn from the errors or not? 
Ask about risk as the innovators see it. Do they 
distinguish between uncertainty (which they do 
not control and wish to minimize) and risk (which 
they can at least manage and calculate)? How 
sustainable is their energy? Above all, look at 
behavior. The innovator’s past actions (i.e., at 
starting things and carrying them to completion) 
are a far more useful predictor than their words 
about the future. 

4. Ensure that money for innovations is 
available on a prompt and flexible basis . On 
the one hand, investors need not have large 
pools of money available to encourage 
innovation. Indeed, more innovation comes from 
bootstrapping rather than from a rich resource 
base. On the other hand, energetic individuals 
with time-bound opportunities cannot wait six 
months for a decision and another six months to 
get funds. And they cannot work with a tightly- 
proscribed line-item budget that cannot be 
adjusted. Contracting guidelines for 
conventional project can stop innovators dead 
in their tracks. 

5. Give as much weight to those innovations 
which are not anchored in theory as to those 
which are. Innovations may well precede 
conceptual development. In technology, for 
example, many of the most important 20th- 
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century innovations began with an individual 
with a hunch, who had the ability to try it out. 
Innovations involve insight and intuition that 
cannot be understood in an analytical context. 
Some are not well understood until completed. 

6. Don \ get hung up on what constitutes an 
innovation and what does not. Definitional 
pursuits of such topics as "newness" quickly 
become red herrings. Instead, structure questions 
so that the response positions the project as an 
innovation in terms of change. Also, be strongly 
guided by projected results. Finally, look for 
characteristics known to be associated with many 
innovations. One is a focus on opportunities, 
not on problems. If a project must be done now 
to capture some time-bound conditions, it is more 
likely an innovation than if it is an idea perennially 
applicable. 

7. Avoid multi-person , multi-criteria review 
processes which are likely to eliminate some of 
the most divergent and promising of 
innovations. One strongly supportive reviewer 
who feels that the innovation deserves a chance 
(for reasons they may or may not fully articulate) 
is worth more than a tepid consensus. The 
sharpest innovations are often eliminated in the 
committee processes. In terms of criteria, 
"reasonableness” and "feasibility” as seen by 
reviewers are not relevant to innovations, even 
if the reviewers are experts in the field. Fred Smith 
received a C from his Ivy League professor for 



the paper that profiled what later become Federal 
Express. The main criticism: his idea was not 
realistic. 

8. Let the innovation be a start-up. Innovators 
need independence, and often cannot start or 
even end within organizational frameworks. As 
long as it is not costly to do so. encourage 
independent initiative. If the innovation does 
take place within an organization, make it a 
condition of your investment that the sponsoring 
group grant the innovation deferment from 
procedural compliance. Further, let the innovator 
operate without monitoring. Use an occasional 
milestone and a focus on results for 
accountability. Remember, the payload is 
information and learning, not service. 

9. Structure your investment to give the 
innovators whatever entrepreneurial framework 
they may need Such a framework may include 
performance-based compensation or bonuses, 
clear personal responsibility, and clear ways to 
keep score. You can also add protection that the 
innovator is locked into the deal. While the 
venture capital approach of asking the 
entrepreneurs to pledge their home and car as 
collateral is not appropriate, innovators may well 
be willing to lend personal resources to the 
enterprise. 

10. Rather than to set aside innovations that 
are deemed too risky ; work with implementors 
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to reduce risk. In many cases, risk is a matter of 
scale and can be handled by reducing the project 
to a more manageable size. Most innovators 
won’t mind. Their interest is in trying out the 
idea and they are surprisingly willing to accept 
compromise as long as something can be tried. 
When the risk is known, ask yourself, "What's 
the worst thing that could happen if this project 
fails?” If you can live with the answer, proceed! 
(At the same time, give no added consideration 
to a project simply because it has a high risk. 
Risk, like novelty, is a side effect of some 
innovations, not a definer of them). 

11. Be prepared to stay with your innovation 
selections once they are made. While many 
projects flourish with funder indifference, 
innovations should not lightly be abandoned. 
Innovators may face unexpected barriers and 
your help in surmounting them can be critical. 
Indeed, some projects should be seen as 
partnerships, in which the investor must be 
prepared to play an active role to deal with 
obstacles that implementors cannot, by 
themselves, overcome. Another form of 
involvement may be progressively greater 
support while a project remains in a fragile start- 
up mode. An initial $ 10.000. three-month project 
may then serve a $50,000. twelve-month project 
in order to work itself into final form and impact. 

12. With the differences discussed, recognize 
that the framework of the full Outcome plan 
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remains relevant . Indeed, the emphasis on a 
tight definition of the product and the connection 
to customers are especially valid with 
innovations. Innovations are often characterized 
by very clear premises, while driven by an 
evident problem of people whose needs are not 
well met by existing products. At the same time, 
not all innovations projects should be required 
to offer great detail. If an innovations project 
had all the answers, it would not be there in the 
first place. 

Where innovation is needed, funders are in the catbird 
seat. More than almost any other group in American life, 
they tend to get what they promote. In may ways, the 
most critical contribution grantmakers can make is actually 
not money. It is a discipline about innovation that 
recognizes that the search for genuine alternatives to 
present practice requires very different tools than the 
support of on-going programs. Innovation is a core 
premise, not an add-on component. Innovation involves 
an explicit test of a new approach, not a massaging of 
dreams or visions. 

Buying change is possible. But one needs the 
coin of that realm. 
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Appemdox 2-11 

Prodmdimg am Impact Upom Commmiuimity Outcomes: 

Implications for Returns on Investment 

About the Appendix: One key is to honestly address what it means to be 



Reforms often seek to think comprehensively, but they 
then act by establishing demonstration programs that are 
limited in scope and potential impact. This excerpt suggests 
the level of investment that may be needed to fully 
implement comprehensive approaches. The annotated 
bibliography provides further information on evaluating 
the potential benefits (returns on investment) from 
comprehensive, community-based reform strategies. 

Source of Excerpt: 

Bruner, Charles, “Where’s the Beef? Getting Bold 
About What 'Comprehensive’ Means,” in Stone, Rebecca 
(cd.) Core Issues in Comprehensive Community-Building 
Initiatives. Chicago, IL: Chapin Hall Center for Children, 
1996. pp. 85-86. 

Excerpt: 

When working within disinvested communities, it is 
important to distinguish between individual and 
community capacity building strategies. Historically, 
individual growth often produces mobility out of poor 
neighborhoods and the two strategies are not necessarily 
synchronous. 

Today, many community initiatives focus upon only 
one of these two important strategies. There clearly is a 
need to better connect the two, conceptually and in 
practice. 
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“comprehensive.” If we limit our thinking to the perceived 
resources at hand, we may fail to explore the true potential 
of a comprehensive approach. 

Unfortunately, few community initiatives today are 
truly “comprehensive.” They lack the resources and 
opportunities to deliver what many socially vulnerable 
families need to grow and develop or the physical and 
economic capital disinvested communities need to rebuild. 
While initiatives may commit to extending beyond a 
categorical focus (e.g. working with families and within 
neighborhoods across multiple dimensions of well-being 
— psychological, social, educational, economic) this does 
not make them “comprehensive.” They still experience 
constraints in providing essential supports. Sometimes, 
families can succeed despite some gaps and services, but 
many times these gaps are too great. Too many families 
are left untouched. 

We do a disservice to suggest that we have any track 
record in testing whether “comprehensive community 
initiatives” can change the lives of residents of 
disinvested neighborhoods and the character of their 
communities. At best, we have been testing whether small- 
scale efforts can establish small footholds and allow some 
individuals the means to escape. 

To be truly comprehensive requires building both 
individual and community capacity in more than a marginal 
way. It requires constructing a critical mass of activities 
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and opportunities so that residents begin to see new 
possibilities both in their own lives and in the lives of 
those around them. Ideally, a comprehensive community 
initiative would build upon itself and become much more 
than the sum of its parts. While mobility will always exist, 
comprehensive initiatives should increase the likelihood 
those experiencing individual growth would stay or that 
individuals with similar aspirations and abilities would 
move in if they did leave. 

Often, existing resource constraints have confined 
our focus. Yet, if we are to find ways to succeed with 
children, families, and neighborhoods that current systems 
fail, we must start by defining the resources we require in 
order to implement at a breadth and depth sufficient to 
substantially improve the life trajectories of community 
residents and the condition of the community in which 
they reside. 

I believe we can make a compelling case for such an 
investment — because the social costs of current failures 
are so great. Leaders in Allegheny County, Pennsylvania, 
identified their fifty-two “highest risk” neighborhoods/ 
where 222.000 of the county's 1,336,000 residents live. 
They are seeking to develop neighborhood-based 
strategies to improve the outcomes for children and families 
residing there. They asked the Child and Family Policy 
Center to establish the potential “return on investment” 
they might expect if they could succeed in reducing high 
rates of “rotten child outcomes” in those neighborhoods 
and their resulting public costs. 

The Child and Family Policy Center contrasted the 
prevalence of “rotten outcomes” (child welfare cases, 
delinquency adjudications, impoverished single parent 
families, and violent crimes) in those highest risk 
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neighborhoods with that in the rest of Allegheny County. 
We used these as proxies for the distribution of public 
expenditures on child welfare, juvenile justice, AFDC, and 
adult imprisonment (we developed other proxies for food 
stamp and Medicaid expenditures and tax revenues). 

We then asked the question, “What if... those highest 
risk neighborhood rates of ‘rotten outcomes’ could be 
reduced to those found in the rest of Allegheny County? 
What would be the public financial gain?” 

The answer was $ 563 million annually. In business 
analogy, there is a potential market of at least $ 563 million 
for successful reforms. The challenge is to develop a 
business plan with a realistic potential to capture a 
sufficient share of that market to justify its cost. 

We can justify an annual investment in the tens, if 
not hundreds, of millions of dollars within these 
neighborhoods if doing so can bring neighborhood 
conditions closer to the rest of Allegheny County. 
Alternatively, even if a small investment, say $1 to $10 
million, is wildly successful and produces a three or five 
to one return, it will scarcely make a dent in the overall 
disadvantage these neighborhoods experience. We can 
afford, and in fact, must (if we are to see clear signs of 
change) be bold in describing the scale that a truly 
comprehensive community approach to change individual 
and community conditions must entail. 

I use the business plan analogy because it does not 
presume that we have proven strategies that merely require 
thoughtful implementation and traditional evaluation. The 
successful entrepreneur is one who sees a market others 
do not and develops new products (often through trial 
and error) to capture it. I believe that is how we must 
approach changes in disinvested neighborhoods. Once 
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we give in to the shape and constraints of the current 
world, we are unlikely to change it. 

Annotated Bibliography — Return on 
Investment Modeling : 

Return on Investment Modeling and Analysis 

Return on Investment (ROI) modeling long has been 
used in the business community to make decisions on 
where to allocate resources. Strategies for assessing the 
ROI of public expenditures on children, youth, and families 
are just emerging. Some have focused upon determining 
the social costs of certain “rotten outcomes 1 ’ (school 
failure, crime and delinquency, substance abuse) and then 
contrasted those with the costs of strategies with some 
evidence of reducing them. Others have examined 
particular strategies, usually fairly robust and 
comprehensive prevention-oriented programs, and sought 
to quantify their demonstrated impacts on the children, 
youth, and families they serve across a number of social 
program expenditures. A few have looked at public 
expenditures on “rotten outcomes 11 systematically, usually 
within specific neighborhoods, to determine the 
investments that would be warranted if they could produce 
system-wide change. Those place-based ROI models that 
have been conducted (Allegheny County in Pennsylvania; 
the Sandtown-Winchester neighborhood in Baltimore. 
Maryland; and Austin, Texas) have helped to identify the 
magnitude of the stakes and the potential investment 
returns involved in transforming neighborhoods. 

There is no single ROI methodology to employ in 
policy making, nor should public commitments to 
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supporting children, youth, and families be evaluated 
simply in terms of public dollar expenditures. Still, ROI 
modeling can provide a discipline for critically examining 
current, as well as proposed, public expenditures in order 
to design more effective strategies to achieve public 
objectives. ROI modeling also can help to make the case 
for investments in strategies at a scale where they can 
begin to impact adverse outcomes and their social costs. 
Finally, ROI modeling may surface strategies for holding 
specific strategies accountable to achieving the results 
they are intended to produce. 

Annotated Bibliography 

I. Theoretical Approaches to Investment-Based 

Budgeting. 

There have been a number of different 
efforts to promote ROI modeling or 
“ cost of failure " analysis as a way to 
re -orient the manner in which state 
and local governments analyze 
expenditures upon children and 
families. The following publications 
present conceptual models rather than 
actual application of the models to 
specific initiatives or financing 
systems. 

The Conservation Company and the Juvenile Law 
Center. Building Bridges: Strategic Planning and 
Alternative Financing for System Reform (Philadelphia, 
PA. 1994). 
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— This monograph offers a conceptual model 
for financing systems conversions, exploring 
several options (such as public debt, internal 
“loan”, funds, and special purpose authorities) 
as a means of financing transition costs. It 
examines two conversions, one in the public 
sector (Broward County, Florida’s juvenile 
detention initiative) and one in the private sector 
(the Saturn Project). 

Bruner. Charles (with Steve Scott). Investment-Based 
Budgeting — The Principles in Converting from a 
Remediation Response to a Prevention/Investment 
Budget (Occasional Paper #11: Child and Family Policy 
Center, Des Moines, IA: 1994). 

— This monograph describes the rationale 
needed to convert from a remediation/response 
to a prevention/investment budgeting approach 
in state government, including the challenges to 
financing the conversion costs. It also offers a 
payback curve for the Perry Pre-School Program 
that shows the relatively long time-horizon 
needed to recapture initial investments when the 
focus is upon prevention programs in the early 
years of life. 

Center for Assessment and Policy Development, 
“Appendix 7: A Primer on Analyzing and Publicizing the 
Cost of Failure,” The Children s Initiative Strategic 
Planning Guide (Bala Cynwyd, PA: 1992). 

— This appendix provided guidance to states 
involved in the Pew Charitable Trusts’ Children’s 
Initiative in how to define the cost of failure (the 
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public costs of current “rotten outcomes” for 
children), think about different audiences for that 
analysis, developing a portfolio of materials 
related to the cost of failure, and publicizing 
results. It represents an early effort to 
incorporate ROI analysis into initiative design 
and thinking. 

Brizius, Jack and the Design Team. Deciding for 
Investment: Getting Returns on Tax Dollars (Alliance for 
Redesigning Government and National Academy of Public 
Administration, Washington, DC: 1994). 

— Written as a workbook, this publication 
provides a very linear approach to using 
outcomes and social costs to make decisions on 
investment strategies and move to financial 
decision-making based upon comparative 
investment values. 

II. Program-Based RO Is. Projecting the Impact of 
Specific Program Interventions on Future Public 
Costs and Spending. 



There have been a number of efforts to 
document the cost-benefits of specific 
programmatic interventions . Some 
have been widely quoted and used to 
justify program expansions — most 
frequently for prenatal care , for 
immunizations , and for early childhood 
education services. For instance , the 
expansion of financial access to 
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prenatal care for low-income women 
often is justified in terms of ax^erted neo- 
natal and other intensive care costs 
and longer-term social costs 
associated with preventable disabling 
conditions ($3.38 saved for every $L00 
invested). 

Schweinhart, Lawrence, Helen Barnes, and David 
Weikart. Significant Benefits: The High/Scope Perry 
Preschool Study Through Age 27 (High/Scope 
Educational Research Foundation: Ypsilanti, MI: 1993). 

— This monograph is based upon a rigorous 
evaluation of a high intensity early childhood 
education program in the early 1 960s, with both 
a treatment and control group tracked over the 
next twenty years. The monograph shows an 
ROI of approximately $7 for every dollar invested, 
with the majority of those savings in reduced 
economic loss to victims of crime as a result of 
lower crime rates among the treatment group and 
other savings in reduced welfare dependency, 
use of special education, and increased earnings 
and tax contributions. Arguably, this research 
has been the most influential ROI analysis in 
securing public investments and has been widely 
cited in state and federal actions to develop and 
expand early childhood programs for 
disadvantaged children. 

Olds, David, et . al “Effects of Prenatal and Infancy 
Nurse Home Visitation on Government Spending,” Medical 
Care Vol. 31, No. 2 (February, 1992), p. 155-174. 
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— This is one of the reports on the Elmira, New 
York project to provide home visiting and 
prevention services to high risk families, showing 
savings across AFDC, Medicaid, food stamps, 
and child protection services that more than 
covered program costs. Olds has written and 
researched extensively on home visiting 
programs, including a meta-analysis that shows 
mixed results for different programmatic efforts. 

Huntington, Jane and Fred Connell, “For Every Dollar 
Spent — The Cost Savings Argument for Prenatal Care,” 
The New England Journal of Medicine, Vol. 331, No. 19 
(Nov. 10, 1994), p 1303-1307. 

— This article reviews the literature on prenatal 
care and found no evidence that initiatives to 
increase prenatal care show immediate savings 
by averting $3 in neonatal expenditures for every 
$1 invested, although they may show other 
benefits. It calls into question simplistic 
approaches to related programs to subsequent 
savings. The actual $3.38 figure, drawn from a 
1985 simulation by the Institute of Medicine, has 
become the rationale for many of the expansions 
of health care coverage for low-income pregnant 
women. 

Select Committee on Children, Youth, and Families. 
Opportunities for Success: Cost-Effective Programs for 
Children Update , 1990 (U.S. Government Printing Office: 
Washington, DC.: 1990). 
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— This report describes a number of programs 
in health, nutrition, and education and draws 
upon the research and evaluation literature to 
argue that each is cost-effective. The dissenting 
views call into question the conclusions of the 
report. The report is a good illustration of how 
claims of cost-effectiveness can go well beyond 
the evaluation and research base to make them. 

Bruner, Charles (with Stephen Scott). Thoughts on 
Statistical and Substantive Significance — Are We Selling 
Programmatic Efforts Short? (Occasional Paper #20: Child 
and Family Policy Center: Des Moines, IA: 1996). 



— This monograph discusses the difficulty in 
constructing sufficiently sensitive measurement 
tools for determining whether or not 
programmatic efforts, particularly those 
individually tailoring responses, achieve their 
goals using experimental research designs. The 
Appendix provides an illustration suggesting that 
programs may have very high ROls across several 
social cost areas although experimental designs 
will not show statistically significant impacts. 

ID. Outcome-Based ROIs. Assessing the Costs of 
Specific Rotten Outcomes/Conditions. 

There have been a number of efforts to 
identify the range of long-term social 
costs associated with certain 
behaviors or conditions , generally 
drawing upon evidence of the 




relationship between a particular 
behavior or condition and subsequent 
“ rotten outcomes '' and their costs. 

These generally have been used to 
make the case for developing strategies 
that can change the behavior or 
conditions. These approaches often 
have been used to gather public 
support for attention to a particular 
issue. Most people have heard of 
studies that cite the costs of alcoholism 
or of smoking in the billions of dollars 
to society annually. 

A. Child abuse. 

Gould, Marsha and Tracey O’Brien. Child 
Maltreatment in Colorado: The Value of Prevention and 
the Cost of Failure to Prevent (Colorado Children’s Trust 
Fund: Denver, CO: 1995). 

— This study develops both direct and indirect 
cost estimates for child abuse and neglect. Direct 
costs include those related to investigating and 
treating abuse and neglect; indirect costs include 
those related to outcomes to which abuse and 
neglect is known to be associated (violence, teen 
pregnancy, domestic violence, criminal behavior, 
substance abuse, mental illness, unemployment, 
welfare dependency, etc.). The study then suggests 
investments that could be made in home visiting 
services to prevent abuse and neglect, with 
projected impacts upon direct and indirect costs. 



Caldwell, Robert. The Costs of Child Abuse vs. Child 
Abuse Prevention: Michigan s Experience . (Children’s 
Trust Fund and Michigan State University : Lansing, MI: 
1992). 



— This study draws from much of the same 
literature as the Gould and O’Brien study. 

B. School failure. 

National Governors’ Association. School Readiness 
(Washington, DC.: 1992). 

— The National Governors’ Association 
commissioned some economic modeling from the 
IBM Customer Business Development Division 
in Boulder, Colorado, on the potential additional 
tax revenues and reduced social spending 
resulting from improving the high school 
graduation rate of at-risk youth from 46% to 65% 
and to 90%. The IBM team also did modeling on 
increased tax revenues from improved higher 
education attainment in Colorado and the 
economic impact of prisoner literacy training 
programs in both increased tax revenues and 
reduced recidivism and social costs. 

C. High risk youth. 

Cohen, Mark. The Monetary Value of Saving a High 
Risk Youth (draft report submitted to the Urban Institute: 
Vanderbilt University. Nashville, TN: 1994). 
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— This report provides an econometric analysis 
of the value of saving a high risk youth from 
three different paths: (1) becoming career 
criminals, (2) becoming heavy drug users, and 
(3) dropping out of school, including a discussion 
of the methodological challenges in aggregating 
across the categories. 

Child and Family Policy Center. Investing in Families , 
Prevention , and School Readiness: A Framework Paper 
(Des Moines, I A: 1993). 

— This paper describes the research and 
evidence on the connection between poor 
outcomes in the early years (poor birth outcomes, 
insufficient nurturing in the early years, and 
inadequate developmental support) and future 
social costs. It then analyzes current public 
spending on prevention-oriented programs and 
contrasts those with social costs related to 
remediation, public protection, and maintenance 
programs for conditions that were at least 
partially preventable in the early years. Finally, it 
suggests the types of investments the state could 
make in avoiding poor outcomes in the early 
years, based upon the numbers of high risk 
families and children and the current investments 
being made in them. 

D. General demographic relationships. 

Haveman, Robert and Barbara Wolfe. Succeeding 
Generations: On the Effects of Investments in Children 
(Russell Sage Foundation: New York. NY. 1994). 
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— This book draws from two decades of 
longitudinal data to evaluate the impact of many 
background factors — including parent 
education, family structure, and neighborhood 
environment — to determine which are most 
strongly associated with child educational 
success, adolescent parenting, and early adult 
employment. It concludes that parental 
educational status and neighborhood 
environment all strongly correlate with child 
educational, social, and economic success. 

IV. Program-Outcome Eased ROIs. Assessingthe 
impact of specific interv entions on specific outcomes. 

There are several recent studies that 
have used ROI modeling approaches 
to assess the relative impacts of 
different responses to specific social 
concerns . In effect , this combines 
program -based ROI modeling with 
outcome-based ROI modeling. Two of 
these have dealt specifically with the 
costs of criminal activity and specific 
programmatic actions to reduce that 
activity 

Greenwood, Peter, Karyn Model, C. Peter Rydell, and 
Jane Chiesa. Diverting Children from a Life of Crime: 
Measuring Costs and Benefits (RAND: Santa Monica. 
CA: 19%). 
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— This sophisticated study employed ROI 
modeling to contrast interventions with 
delinquent youth (both intensive monitoring and 
educational scholarships) with “three strikes and 
you're out” legislation, concluding the former 
had higher ROIs in averted criminal justice costs. 
Early childhood services had very low ROIs under 
the modeling, but only averted criminal justice 
costs were considered as social cost impacts. 

Piehl, Anne and John Dilulio, Jr., “'Does Prison Pay?’ 
Revisited,” The Brookings Review (Winter, 1995), p. 21- 
25. 

— Using a variety of techniques, this article 
concludes that the specific incapacitation value 
of incarceration is cost-effective for dangerous, 
repeat offenders but not for those involved in 
drug trafficking. 

V. Place-Based ROIs. Examining Social Costs (and 
Investment Opportunities) in Oisinvested 
Neighborhoods. 

There are at least three examples of 
ROI modeling efforts applied to specific 
disinvested neighborhoods 

experiencing high levels of ” rotten 
outcomes ” and subsequent social costs 
(e.g. unemployment and welfare 
dependency and loss of tax base , crime 
and delinquency , social remediation 
expenditures in health and child 
welfare). This approach recognizes the 
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interrelationship among different 
factors that cause “ rotten outcomes ” 
and the “ rotten outcomes " they 
produce. 

A . Austin, Texas . 

The Austin Project. ^/7 Investment Plan For The 
Young: The Austin Project, First Phase (Austin, TX: 1992). 

— Employing geo-mapping, the Austin Project 
distinguished between thedntensity of problems 
faced in Austin’s inner-city areas and those faced 
in other parts of the community and began to 
quantify the systemic benefits of reducing those 
problems, as well as suggesting the potential 
cost-effectiveness of investments in certain 
prevention programs. 

B. Sandtown-Winchester , Baltimore , Maryland 

The Enterprise Foundation. An Economic Model for 
the Transformation of Sand town- Winchester: Discussion 
of Methodology and Supporting Data (Baltimore, MD. 
19%). 

The Enterprise Foundation. Neighborhood 
Transformation Investment Plan (Baltimore, MD: 1996). 

Downs, Anthony. Observations on the Enterprise 
Foundation s Project in Baltimore s Sand town - 
Winchester Area. Memorandum to the Enterprise 
Foundation, n.d. 



O 



Appendices 



— The residents of Sandtown-Winchester in 
Baltimore, with assistance from the Enterprise 
Foundation, are seeking to transform the entire 
neighborhood. As one aspect of this work, the 
Enterprise Foundation developed an assessment 
of public costs incurred within Sandtown- 
Winchester, a “business as usual" trend-line on 
these costs, and the types of investments that 
could be made that would alter this trend-line, 
showing a positive, overall return-on-investment 
through the year 2015. 

— Anthony Downs reviewed the methodology 
and work of the Enterprise Foundation and 
provided a critique of this work, accepting the 
use of “simple-minded quantification" as an 
appropriate tool for demonstrating the 
dimensions of the issues, citing some of the 
limitations of applying internal rate of return 
analysis to complex public systems, and raising 
issues of the application of the ROI modeling to 
influence funding decisions. 

C. High Risk Neighborhoods in Allegheny 
County, Pennsylvania . 

Bruner, Charles (with Stephen Scott and Martha 
Steketee). Background Paper, A lleghenv County Study: 
Potential Returns on Investment from a Comprehensive 
Family Center Approach in High-Risk Neighborhoods 
(Child and Family Policy Center: Des Moines. I A: 1996). 
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— This monograph employs geo-mapping to 
contrast the prevalence of “rotten outcomes” in 
Allegheny County’s highest risk neighborhoods 
with the remainder of the county and then uses 
these as proxies for public spending in six areas 
(AFDC food stamps, Medicaid, child welfare, 
juvenile detention, and adult prisons). It 
describes the potential market for new 
investments as the amount of reduced public 
funding and increased tax revenues available, if 
these neighborhoods were to be transformed to 
bear the characteristics present in the rest of 
Allegheny County. Finally, it does some modeling 
with a specific investment strategy — Family 
Centers — to suggest its potential impact upon 
these outcomes. 

VL Place-Based Theories: Rationales for Investment 

in Disinvested Neighborhoods 

It is one thing to examine current and 
projected costs associated with the 
“ rotten outcomes ” prevalent in 
disinvested neighborhoods . It is 
another thing to connect programs and 
strategies to changing those “rotten 
outcomes . ” The following are several 
different explorations into the potential 
for comprehensive approaches to 
“succeed within neighborhoods current 
systems fail, ” in effect constructing a 
theory for change within disinvested 
neighborhoods . 
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Porter, Michael, “The Competitive Advantage of the 
Inner City,” Hansard Business Review (May- June, 1995), 
p. 55-71. 



— Porter argues that the solutions to the problems 
of the inner-city must be based upon fostering 
inner city business development rather than 
increasing social investment and hoping for 
economic activity to follow. He poses specific 
advantages to an inner city development 
strategy: strategic location, local market demand, 
integration with regional clusters, and human 
resources. 

Lemann, Nicholas, “The Myth of Community 
Development,” New York Times Sunday Magazine 
(January, 1994), p. 27-3 Iff. 

— In this and other work, Lemann critiques efforts 
to rebuild neighborhoods, in part because he 
argues it may be impossible to truly regenerate 
neighborhoods, as residents who experience 
personal economic growth move out. 

Research and Policy Committee of the Committee for 
Economic Development. Rebuilding Inner-Citv 
Communities: A New Approach to the Nation s Urban 
Crisis . 



— This influential committee of business and 
education leaders promotes the development of 
partnerships between government and business 
in revitalizing distressed neighborhoods. It draws 
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from some of the cost-benefit literature on 
specific programmatic interventions in justifying 
such investments. 

Bruner. Charles. Realizing a Vision for Children, 
Families , and Neighborhoods: An Alternative to Other 
Modest Proposals (National Center for Service Integration 
Clearinghouse: Dcs Moines. IA: 1996). 

— This monograph reviews a diverse array of 
research and theory on the conditions needed 
for children to succeed at high levels, concluding 
that it will require a critical mass of family support, 
systems reform, social capital development, and 
community economic development within 
disinvested neighborhoods to alter the 
expectations for children and families, but that 
the stakes involved make the effort worthwhile. 

Bruner, Charles. “How Can Empowerment Zones 
Succeed? The Case of the ‘NotSoEZ’ Community, in 
Disinvested Communities and Comprehensive Sendees: 
Linking Family ; Community, and Economic Development. 
(Occasional Paper #15: Child and Family Policy Center, 
Des Moines. 1A: 1995). 

— This article describes the characteristics of an 
inner-city neighborhood in terms of job readiness 
and job needs. It then suggests the types of 
human capital development strategies necessary 
for the population to be job ready, and the mix of 
public sector, indigenous, and external private 
sector jobs that residents require to transform. 
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Appendix 2=12 

Measuring System Accountability 



About this Appendix: 

Each service system must have its own set of 
accountability measures, based upon its role in producing 
the conditions needed to achieve community outcomes. 
This excerpt from a longer monograph by Charles Bruner 
describes, first gcnerically and then applied to two service 
systems, the characteristics of "‘productive service 
systems." 

Source: 

Bruner. Charles. A Framework for Developing and 
Holding Comprehensive Reform Efforts Accountable for 
Improving Child Outcomes. Des Moines, I A: Child and 
Family Policy Center, 1995. 

Excerpt: 

System outcomes. A third necessary ingredient to 
improving long-term outcomes relates to how the many 
public and professional service systems serving children 
and families respond to their needs. While each may have 
a primary objective (public schools in educating youth; 
health clinics in providing medical services; substance 
abuse treatment programs in helping individuals overcome 
chemical dependency), they cannot be isolated from other 
systems and must be connected both to effective frontline 
practices and to whole villages They must develop a 
connection with the children and families they serve and, 
in turn, help strengthen child and family bonds with the 
whole village. 



The Figure below provides a listing of six attributes 
of productive service systems — applicable both to 
individual services serving children and families and to 
the configuration of these services as a whole. While the 
specific measures for each may differ, depending upon 
whether one is examining child welfare services or looking 
at K-12 schooling, measures can be constructed to provide 
a reasonably complete view of whether the systems are 
producing these results. 



llHItllllg - -Vrflfflire • 

AttribttjKJs of Prodweitiw : ■ 

• Seamless and Coordinated in Response 
9 Integrated into Community Life 
• Timely in Responding to Opportunities 
9 Sufficient to Meet Objectives 
9 Efficient in Meeting Objectives 
° Adaptive and Learning 

In this figure, seamless and coordinated in response 
refers to the need for individual systems to have effective 
referrals and follow-ups to other systems, when children 
or families require services and supports that require that 
outside response. It also speaks to minimizing the 
disruptions and discontinuities that can occur in such 
transfers, and pays particular attention to making smooth 
“hand-offs" from one service to another This 
seamlessness also must exist with respect to new frontline 



approaches being undertaken to help families in their 
growth. In many instances, the professional service 
systems must be in a responsive position to those frontline 
workers and their families. 

Integrated into community life means that services 
seek, as much as possible, to create pathways for children 
and families to voluntary and community support systems, 
as opposed to ongoing professional services. When 
ongoing professional services are needed, the emphasis 
is still upon providing as much support as possible within 
the child and family’s own community circle. Professional 
systems have a responsibility to connect the child, youth, 
or family to the ‘"whole village.” 

Timely in responding to opportunities means that 
services are sufficiently flexible to respond when needed. 
Part of being seamless in delivery of services and supports 
is being timely. In addition, however, productive systems 
take advantage of the moment of opportunity. Time often 
is used as a management tool to reduce the demand on the 
system; productive systems can identify when immediate 
response is needed and take action accordingly. 

Sufficient to meet objectives means that sufficient 
resources are devoted to achieve the system’s specific 
objectives and that services are not diluted to the point 
that they are ineffective. The objectives of any particular 
system may represent only a portion of the actions needed 
to produce an impact on long-term outcomes, but 
resources must be sufficient to meet those objectives. 

Efficient in meeting objectives means that there is a 
minimum of wasted effort and that resources are focused 
on the task at hand. 

Adaptive and learning means that services 
continue to use their experiences to improve their capacity 
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to respond effectively. Citizen monitoring and parent 
feedback mechanisms constitute practices that can assist 
in assuring that services learn from the consumer’s, as 
well as their own, experiences. 

The following two figures apply these abstract 
attributes to two very different, but important parts of a 
community's system of services for children and families 
— child welfare and elementary education. In both 
instances, the measures developed correspond to the 
system’s achievement of these goals in practice, based 
upon broad system objectives. 



fBHBl Chilli 

Seamless and Coordinated in Response 

•System minimizes reliance upon remote, 
institutional, and out-of-home care. 

•System minimizes the number of moves of a child 
from one setting to another, and, when moves 
are necessary, transfer is made in a fashion that 
all relevant information is transferred. 

•System identifies child’s special educational, 
health, and social support needs and secures 
needed responses from those systems to meet 
the needs, minimizing discontinuities and 
movements from one system to another. 

Integrated Into Community Life 

•Placements are made with the least disruption 
possible, with maintenance of frequent contact 
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with parents and siblings, except where 
unwarranted. 

•When placements are made, dislocations from 
family, supportive relatives, schools, and positive 
peer environments are minimized. 

•System takes steps to establish contacts and 
ties for children, families, and caregivers within 
the community, extending beyond professional 
service systems. , 

Timely in Responding to Opportunities 

•System establishes realistic permanency plan 
quickly and engages all parties (especially 
parents) in defining and fulfilling their roles and 
responsibilities under the plan. 

•Crises are responded to immediately, without 
resorting to placement as a vehicle for “ sorting 
things out. 1 ' 

Sufficient to Meet Objectives 

•System achieves permanency plan goals in 
reasonable time frames. 

Efficient in Meeting Objectives 

•System minimizes the use of institutional and 
residential placements and reduces the length of 
stay in placement. 

•System protects the safety of the child and 
minimizes occurrence or re-occurrence of abuse 
or neglect (both in family and in substitute care). 

Adaptive and Learning 

•New information and incidents inform 
permanency plan goals and objectives. 





Seamless and Coordinated in Response 

•Schools identify non-educational needs of 
children and families and secure needed 
responses from other systems to meet those non- 
educational needs. 

•Schools follow-up with those systems to assure 
that responses and strategies complement and 
are coordinated with school activities with the 
child (and family). 

Integrated Into Community Life 

•Parents regard the school as a resource and 
support to them and their children. 

•Parents are involved in school and their 
children's education. 

•Schools have knowledge of, and maintain and 
relationships with, other voluntary community 
organizations and institutions that enable 
appropriate schools to serve as referrals to or 
recruiters for those voluntary organizations. 

Timely in Responding to Opportunities 

•Schools respond immediately to absences from 
school or changes in a child’s school behavior 
and performance, in sufficient depth to identify 
and respond to underlying reasons for those 
changes. 



Sufficient to Meet Objectives 

•Children receive the instructional support 
needed to master age-appropriate skills and 
proceed on-grade at reasonable levels of 
achievement. 

Efficient in Meeting Objectives 

•Schools minimize the transfer of children outside 
the classroom and outside the school for meeting 
need. 

•Schools make extensive use of volunteers and 
parental participation in their children’s learning. 

Adaptive and Learning 

•Schools build into their planning systems 
feedback and perspectives from a diverse array 
of parents and students, and decisions reflect 
this feedback. 
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Other NCSI Publications 



Resource Briefs 

• #1 So You Think You Need Some Help? Making Effective Use of Technical 
Assistance, 1992. 

• #2 Charting a Course: Assessing a Community’s Strengths and Needs, 1 993. 

• #3 Who Should Know What: Confidentiality and Information Sharing in 
Service Integration, 1993. 

• #4 Getting to the Bottom Line: State and Community Strategies for Financing 
Comprehensive Community Service Systems, 1 993. 

• #5 Getting Started: Planning a Comprehensive Service Initiative, 1 994. 

• #6 Making it Simpler: Streamlining and Integrating Intake Eligibility, 1 993. 

• #7 Making a Difference: Moving to Outcome-Based Accountability for 
Comprehensive Service Reforms, 1 994. 



Working Papers 

• Beyond Buzzwords: Key Principles in Effective Frontline Practice, 1 994. 

• Steps Along an Uncertain Path: State Initiatives Promoting Comprehensive, 
Community-Based Reform, 1 996. 

• Realizing a Vision for Children, Families and Neighborhoods: An Alternative to 
Other Modest Proposals, 1 996. 

The above publications are available for $4.00 each prepaid from the Child and Family 
Policy Center, 21 8 Sixth Avenue, Suite 1021, DesMoines, IA50309-4006, Tel: (515) 
280-9027, Fax: (515)244-8997. Iowa orders add 5% state sales tax. 
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Matter of Commitment 

NCSI Clearinghouse Guidebooks 



Guidebook 1 : Understanding the Big Picture: Deveioping a Strategic Approach to Reform 

Shows the interconnectedness of the different Guidebooks and provides a framework for taking action. 

Guidebook 2*: Defining the Prize: From Agreed-Upon Outcomes to Results-Based Accountability. 

Describes the manner in which communities can establish measurable goals and how these relate to programmatic strategies and accountability. 

Guidebook 3*: Valuing Diversity and Practicing Inclusion: Core Aspects of Collaborative Work . 

Discusses the importance of recognizing differences and placing issues of race, class, and culture on the table. 

Guidebook4: Where the Rubber Meets the Road: Constructing Effective Services and Supports. 

Outlines the different changes needed in health, education, and human service systems, including the development of new preventive systems. 

Guidebook 5*: Creating Opportunity: Making the Link to Housing , Jobs , and Economic Development . 

Discusses how service collaborations can contribute to addressing neighborhood and community economic needs. 

Guidebook 6*: Getting to the Grassroots: Neighborhood Organizing and Mobilization. 

Describes the steps that service collaborations must take to connect with neighborhood resources, reach out to individuals, and include 
neighborhood voices in all aspects of reform. 

Guidebook 7: From Recipient to Contributor: Parent and Youth Involvement in Decision-Making and Service Delivery. 

Describes how to engage youth and families at both the service delivery and the policy development levels. 

Guidebook 8: Gaining and Exercising Authority: Building Local Decision-Making and Governance Structures. 

Examines different approaches to creating decision-making structures at the community level which are sustainable, representative, legitimate, 
and capable of marshalling resources across systems to achieve agreed-upon goals. 

Guidebook 9: Rethinking Financing: Moving From Funding Silos and Toward Investment-Based Budgeting. 

Describes strategies to ensure financing systems are linked to reform goals and accountable to achieving desired results for children, families, 
and neighborhoods. 

Guidebook 10: The Road to Success: Building the Capacity to Manage Change. 

Describes investments in leadership development and organizational change strategies that can create the capacity to implement reforms. 

Guidebook 1 1 : Delivering on the Vision: Tools and Strategies for Frontline Professional Development . 

Describes approaches for building the skills and qualities needed for changing worker roles at the frontline and supervisory levels. 

Guidebook 12: Building Public Will and Commitment to Sustain the Work. 

Describes ways to build broad public understanding and support. 

Guidebook 13*: Learning from Doing: Continuous Evaluation and Quality Improvement. 

Introduces approaches to evaluation that recognize the path-breaking work community collaborations undertake. 

Guidebook 1 4: Going to Scale: Broadening and Deepening the Commitment to Success. 

Describes the importance and challenge of extending beyond demonstration efforts to changed systems of services and supports. 



* Guidebooks currently published and available 
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